[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 18:59:54 dist-test-slave-49bz krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 18:59:54 dist-test-slave-49bz krb5kdc[399](info): set up 2 sockets
Sep 25 18:59:54 dist-test-slave-49bz krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 18:59:55.018318 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T18:59:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T18:59:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 18:59:55.122666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 18:59:55.197820 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 18:59:55.198047 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 18:59:55.198082 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 18:59:55.199247 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 18:59:55.199316 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 18:59:55.199340 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 18:59:55.199369 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 18:59:55.199395 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 18:59:55.200819 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 18:59:55.201117 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 18:59:55.201349 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:59:55.206128 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:55.206173 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:55.206336 412 server_base.cc:1034] running on GCE node
W20260925 18:59:55.206404 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:55.206609 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 18:59:55.206890 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 18:59:55.208063 412 hybrid_clock.cc:648] HybridClock initialized: now 1790362795208042 us; error 38 us; skew 500 ppm
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 18:59:55.210636 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 18:59:55.211045 412 webserver.cc:460] Webserver started at http://127.0.98.62:44415/ using document root <none> and password file <none>
I20260925 18:59:55.211236 412 fs_manager.cc:359] Metadata directory not provided
I20260925 18:59:55.211279 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:59:55.211424 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:59:55.212391 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "f8b262d582ac4f28936ba56618c450db"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "7a2d43742ceca8ba62ad774d9bc3a639"
server_key_iv: "d119dcffacca4537cb5eb9e8ab4d94b4"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.212731 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8b262d582ac4f28936ba56618c450db"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "7a2d43742ceca8ba62ad774d9bc3a639"
server_key_iv: "d119dcffacca4537cb5eb9e8ab4d94b4"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.213997 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 18:59:55.214660 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.214859 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.214941 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "f8b262d582ac4f28936ba56618c450db"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "7a2d43742ceca8ba62ad774d9bc3a639"
server_key_iv: "d119dcffacca4537cb5eb9e8ab4d94b4"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.215000 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:59:55.223932 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:59:55.225451 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 18:59:55.225596 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:59:55.229203 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37993
I20260925 18:59:55.229240 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37993 every 8 connection(s)
I20260925 18:59:55.229523 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 18:59:55.230368 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:59:55.232254 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db: Bootstrap starting.
I20260925 18:59:55.232728 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db: Neither blocks nor log segments found. Creating new log.
I20260925 18:59:55.232949 480 log.cc:824] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db: Log is configured to *not* fsync() on all Append() calls
I20260925 18:59:55.233529 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db: No bootstrap required, opened a new log
I20260925 18:59:55.235034 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b262d582ac4f28936ba56618c450db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260925 18:59:55.235147 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:59:55.235167 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b262d582ac4f28936ba56618c450db, State: Initialized, Role: FOLLOWER
I20260925 18:59:55.235296 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b262d582ac4f28936ba56618c450db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260925 18:59:55.235370 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:59:55.235392 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:59:55.235417 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:59:55.235880 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b262d582ac4f28936ba56618c450db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260925 18:59:55.235953 480 leader_election.cc:302] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8b262d582ac4f28936ba56618c450db; no voters:
I20260925 18:59:55.236085 480 leader_election.cc:288] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:59:55.236500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 412
I20260925 18:59:55.236634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 18:59:55.236145 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:59:55.236301 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:59:55.236850 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [term 1 LEADER]: Becoming Leader. State: Replica: f8b262d582ac4f28936ba56618c450db, State: Running, Role: LEADER
I20260925 18:59:55.236941 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b262d582ac4f28936ba56618c450db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260925 18:59:55.237403 392 external_mini_cluster.cc:1428] Setting key 5007695e06c6829048875d67b1e98c13
I20260925 18:59:55.237434 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8b262d582ac4f28936ba56618c450db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b262d582ac4f28936ba56618c450db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } }
I20260925 18:59:55.237525 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [sys.catalog]: This master's current role is: LEADER
I20260925 18:59:55.237702 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b262d582ac4f28936ba56618c450db. Latest consensus state: current_term: 1 leader_uuid: "f8b262d582ac4f28936ba56618c450db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b262d582ac4f28936ba56618c450db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } }
I20260925 18:59:55.237767 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db [sys.catalog]: This master's current role is: LEADER
I20260925 18:59:55.237838 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:59:55.238474 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:59:55.240163 492 catalog_manager.cc:1347] Generated new cluster ID: 280bac9ef8e048c1bc8339912fae90cf
I20260925 18:59:55.240247 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 18:59:55.248301 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:59:55.248864 492 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:59:55.251281 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:55.238315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.0.98.62:37993)
0925 18:59:55.239047 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:55.239055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:55.239086 (+ 31us) server_negotiation.cc:407] Connection header received
0925 18:59:55.240862 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:55.240867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:55.240904 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:55.241041 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:55.242420 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.243296 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.244256 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.244656 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.244786 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:55.247872 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:55.247876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:55.247876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:55.247890 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:55.249523 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.249940 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.249943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.249943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.250023 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.250297 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.250298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.250298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.250477 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:55.250640 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:55.250906 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 18:59:55.251046 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":69,"threads_started":1}
I20260925 18:59:55.258965 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8b262d582ac4f28936ba56618c450db: Generated new TSK 0
I20260925 18:59:55.259123 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 18:59:55.284238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 18:59:55.383167 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 18:59:55.383337 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 18:59:55.383358 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 18:59:55.384384 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 18:59:55.384421 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 18:59:55.384446 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 18:59:55.385751 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 18:59:55.385998 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 18:59:55.386220 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:59:55.391012 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:55.391026 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:55.391026 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:55.391283 506 server_base.cc:1034] running on GCE node
I20260925 18:59:55.391454 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 18:59:55.391670 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 18:59:55.392827 506 hybrid_clock.cc:648] HybridClock initialized: now 1790362795392814 us; error 29 us; skew 500 ppm
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 18:59:55.395165 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 18:59:55.395605 506 webserver.cc:460] Webserver started at http://127.0.98.1:44371/ using document root <none> and password file <none>
I20260925 18:59:55.395820 506 fs_manager.cc:359] Metadata directory not provided
I20260925 18:59:55.395884 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:59:55.396025 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:59:55.396984 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "94a66018cf6241b0a49e87c4877801b5"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "819b9d7b446fb72de980580ffd752df2"
server_key_iv: "04f16392179ce95873ac49bca44a82e2"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.397343 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "94a66018cf6241b0a49e87c4877801b5"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "819b9d7b446fb72de980580ffd752df2"
server_key_iv: "04f16392179ce95873ac49bca44a82e2"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.398736 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 18:59:55.399463 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.399631 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 18:59:55.399724 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "94a66018cf6241b0a49e87c4877801b5"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "819b9d7b446fb72de980580ffd752df2"
server_key_iv: "04f16392179ce95873ac49bca44a82e2"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.399806 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:59:55.408993 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:59:55.410766 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 18:59:55.410907 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:59:55.411121 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:59:55.411427 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:59:55.411476 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.411514 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:59:55.411540 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.417158 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38529
I20260925 18:59:55.417209 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38529 every 8 connection(s)
I20260925 18:59:55.417544 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 18:59:55.418643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 506
I20260925 18:59:55.418753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 18:59:55.418895 392 external_mini_cluster.cc:1428] Setting key abb1b7516e459d07c3aa7225d75f07d8
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 18:59:55.427491 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:55.418198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47229 (local address 127.0.98.62:37993)
0925 18:59:55.418309 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:55.418312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:55.418522 (+ 210us) server_negotiation.cc:407] Connection header received
0925 18:59:55.419031 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:55.419032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:55.419045 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:55.419109 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:55.420119 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.420876 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.422188 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.422535 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.422605 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:55.424809 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:55.424810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:55.424811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:55.424817 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:55.426293 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.426678 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.426680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.426681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.426739 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.426984 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.426985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.426985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.427090 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:55.427145 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:55.427365 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 18:59:55.427431 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 18:59:55.427630 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:55.418201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.1:47229)
0925 18:59:55.418374 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:55.418411 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:55.418929 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:55.419175 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:55.419179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:55.419778 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 18:59:55.420021 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:55.420028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.420982 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.420983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:55.422102 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:55.422104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.422615 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.422616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:55.422721 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:55.423230 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 18:59:55.423238 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 18:59:55.424716 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 18:59:55.426371 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:55.426377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:55.426378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:55.426584 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:55.426809 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:55.426810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:55.426810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:55.426898 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:55.427148 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 18:59:55.427152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 18:59:55.427250 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 18:59:55.427334 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 18:59:55.427439 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260925 18:59:55.428086 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260925 18:59:55.428174 635 heartbeater.cc:464] Registering TS with master...
I20260925 18:59:55.428360 635 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 18:59:55.428962 444 ts_manager.cc:194] Registered new tserver with Master: 94a66018cf6241b0a49e87c4877801b5 (127.0.98.1:38529)
I20260925 18:59:55.429515 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 18:59:55.442602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 18:59:55.515971 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 18:59:55.516144 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 18:59:55.516165 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 18:59:55.517351 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 18:59:55.517388 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 18:59:55.517441 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 18:59:55.519027 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 18:59:55.519253 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 18:59:55.519464 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:59:55.524130 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:55.524186 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:55.524286 640 server_base.cc:1034] running on GCE node
W20260925 18:59:55.524156 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:55.524561 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 18:59:55.524804 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 18:59:55.525934 640 hybrid_clock.cc:648] HybridClock initialized: now 1790362795525927 us; error 25 us; skew 500 ppm
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 18:59:55.528234 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 18:59:55.528631 640 webserver.cc:460] Webserver started at http://127.0.98.2:43213/ using document root <none> and password file <none>
I20260925 18:59:55.528841 640 fs_manager.cc:359] Metadata directory not provided
I20260925 18:59:55.528880 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:59:55.529012 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:59:55.530200 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "03a808c9df224c97931c2ebe26ccb548"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "e0801a52735f036b9ce0ce11ec607807"
server_key_iv: "5278fc641e74415960b5378546362a57"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.530617 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "03a808c9df224c97931c2ebe26ccb548"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "e0801a52735f036b9ce0ce11ec607807"
server_key_iv: "5278fc641e74415960b5378546362a57"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.531865 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 18:59:55.532512 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.532691 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 18:59:55.532787 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "03a808c9df224c97931c2ebe26ccb548"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "e0801a52735f036b9ce0ce11ec607807"
server_key_iv: "5278fc641e74415960b5378546362a57"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.532866 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:59:55.558404 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:59:55.560139 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 18:59:55.560287 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:59:55.560488 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:59:55.560773 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:59:55.560828 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.560858 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:59:55.560884 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.566705 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34459
I20260925 18:59:55.566820 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34459 every 8 connection(s)
I20260925 18:59:55.567130 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 18:59:55.575450 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:55.567812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54905 (local address 127.0.98.62:37993)
0925 18:59:55.567855 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:55.567857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:55.568057 (+ 200us) server_negotiation.cc:407] Connection header received
0925 18:59:55.568557 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:55.568558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:55.568581 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:55.568654 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:55.569561 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.570053 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.571053 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.571230 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.571301 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:55.573130 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:55.573131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:55.573131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:55.573138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:55.574223 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.574625 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.574626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.574626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.574678 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.574900 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.574900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.574901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.574996 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:55.575063 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:55.575298 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 18:59:55.575367 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 18:59:55.575613 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:55.567777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.2:54905)
0925 18:59:55.567939 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:55.567979 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:55.568458 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:55.568735 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:55.568739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:55.569288 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 18:59:55.569463 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:55.569467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.570151 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.570152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:55.570971 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:55.570973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.571305 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.571306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:55.571423 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:55.571813 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 18:59:55.571820 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 18:59:55.573034 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 18:59:55.574350 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:55.574356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:55.574358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:55.574548 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:55.574736 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:55.574737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:55.574737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:55.574835 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:55.575055 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 18:59:55.575060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 18:59:55.575184 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 18:59:55.575286 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 18:59:55.575419 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260925 18:59:55.576040 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260925 18:59:55.576102 769 heartbeater.cc:464] Registering TS with master...
I20260925 18:59:55.576252 769 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260925 18:59:55.576578 444 ts_manager.cc:194] Registered new tserver with Master: 03a808c9df224c97931c2ebe26ccb548 (127.0.98.2:34459)
I20260925 18:59:55.576767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 640
I20260925 18:59:55.576864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 18:59:55.577005 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54905
I20260925 18:59:55.577041 392 external_mini_cluster.cc:1428] Setting key caaa307859752941b6cae43bc64a522d
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 18:59:55.598835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 18:59:55.674947 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 18:59:55.675149 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 18:59:55.675187 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 18:59:55.676203 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 18:59:55.676262 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 18:59:55.676303 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 18:59:55.677649 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 18:59:55.677901 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 18:59:55.678105 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:59:55.682621 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:55.682654 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:55.682917 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:55.683120 774 server_base.cc:1034] running on GCE node
I20260925 18:59:55.683301 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 18:59:55.683516 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 18:59:55.684679 774 hybrid_clock.cc:648] HybridClock initialized: now 1790362795684658 us; error 33 us; skew 500 ppm
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 18:59:55.687135 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 18:59:55.687531 774 webserver.cc:460] Webserver started at http://127.0.98.3:34519/ using document root <none> and password file <none>
I20260925 18:59:55.687752 774 fs_manager.cc:359] Metadata directory not provided
I20260925 18:59:55.687824 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:59:55.687970 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:59:55.688910 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "63ab932674574af2b0f75d5afafe00ec"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "5b7c652f4fc2c428ce513d1854784127"
server_key_iv: "2a6465524bc53cd5a7329afa5415a2df"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.689245 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "63ab932674574af2b0f75d5afafe00ec"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "5b7c652f4fc2c428ce513d1854784127"
server_key_iv: "2a6465524bc53cd5a7329afa5415a2df"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.690544 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 18:59:55.691288 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.691469 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 18:59:55.691567 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "63ab932674574af2b0f75d5afafe00ec"
format_stamp: "Formatted at 2026-09-25 18:59:55 on dist-test-slave-49bz"
server_key: "5b7c652f4fc2c428ce513d1854784127"
server_key_iv: "2a6465524bc53cd5a7329afa5415a2df"
server_key_version: "encryptionkey@0"
I20260925 18:59:55.691640 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:59:55.707692 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:59:55.709882 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 18:59:55.710044 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:59:55.710240 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:59:55.710573 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:59:55.710621 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.710657 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:59:55.710681 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:55.716502 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44543
I20260925 18:59:55.716547 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44543 every 8 connection(s)
I20260925 18:59:55.716857 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 18:59:55.722968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 774
I20260925 18:59:55.723069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 18:59:55.723260 392 external_mini_cluster.cc:1428] Setting key 71564f0565e8ee02e47b17327e526b0d
I20260925 18:59:55.725580 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:55.717534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59745 (local address 127.0.98.62:37993)
0925 18:59:55.717600 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:55.717602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:55.717787 (+ 185us) server_negotiation.cc:407] Connection header received
0925 18:59:55.718357 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:55.718358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:55.718370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:55.718425 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:55.719279 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.719768 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.720709 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.720889 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.720963 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:55.722793 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:55.722794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:55.722794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:55.722799 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:55.724519 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.724907 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.724907 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.724908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.724956 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.725156 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.725156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.725157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.725253 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:55.725296 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:55.725480 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 18:59:55.725527 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 18:59:55.725757 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:55.717527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.3:59745)
0925 18:59:55.717683 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:55.717716 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:55.718262 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:55.718490 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:55.718493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:55.719008 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 18:59:55.719163 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:55.719167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.719858 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.719859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:55.720624 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:55.720626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.720970 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.720971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:55.721074 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:55.721506 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 18:59:55.721513 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 18:59:55.722717 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 18:59:55.724593 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:55.724598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:55.724600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:55.724825 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:55.725023 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:55.725024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:55.725024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:55.725107 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:55.725302 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 18:59:55.725305 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 18:59:55.725399 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 18:59:55.725469 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 18:59:55.725568 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260925 18:59:55.726166 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260925 18:59:55.726290 903 heartbeater.cc:464] Registering TS with master...
I20260925 18:59:55.726447 903 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260925 18:59:55.726720 444 ts_manager.cc:194] Registered new tserver with Master: 63ab932674574af2b0f75d5afafe00ec (127.0.98.3:44543)
I20260925 18:59:55.727069 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59745
I20260925 18:59:55.735330 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 18:59:55.756242 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:55.749199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.0.98.1:38529)
0925 18:59:55.749352 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:55.749356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:55.749366 (+ 10us) server_negotiation.cc:407] Connection header received
0925 18:59:55.749479 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:55.749481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:55.749494 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:55.749582 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:55.750128 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.750864 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.751726 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.752014 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.752061 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:55.753899 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:55.753905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:55.753905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:55.753910 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:55.755350 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.755629 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.755631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.755632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.755678 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.755823 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.755823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.755823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.755920 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:55.756010 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:55.756064 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 18:59:55.756117 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
W20260925 18:59:55.756584 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52594
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 18:59:55.776342 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:55.770454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.0.98.1:38529)
0925 18:59:55.770516 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:55.770518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:55.770642 (+ 124us) server_negotiation.cc:407] Connection header received
0925 18:59:55.770709 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:55.770710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:55.770722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:55.770784 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:55.771285 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.771705 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.772330 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:55.772508 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:55.772573 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:55.774358 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:55.774359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:55.774359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:55.774364 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:55.775525 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.775821 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.775822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.775822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.775866 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:55.776016 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:55.776016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:55.776016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:55.776105 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:55.776180 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:55.776240 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 18:59:55.776291 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 18:59:55.776786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 506
I20260925 18:59:55.781163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 640
I20260925 18:59:55.786193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 774
I20260925 18:59:55.789985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 412
2026-09-25T18:59:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1750 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[917](info): set up 2 sockets
Sep 25 18:59:55 dist-test-slave-49bz krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260925 18:59:56.793704 392 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 18:59:56 dist-test-slave-49bz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T18:59:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T18:59:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 18:59:56.899284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37579
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 18:59:56.973558 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 18:59:56.973801 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 18:59:56.973836 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 18:59:56.974901 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 18:59:56.974969 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 18:59:56.974992 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 18:59:56.975008 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 18:59:56.975035 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 18:59:56.976365 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 18:59:56.976619 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 18:59:56.976840 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:59:56.981482 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:56.981508 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:56.981508 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:56.981710 930 server_base.cc:1034] running on GCE node
I20260925 18:59:56.981933 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 18:59:56.982197 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 18:59:56.983331 930 hybrid_clock.cc:648] HybridClock initialized: now 1790362796983311 us; error 35 us; skew 500 ppm
Sep 25 18:59:56 dist-test-slave-49bz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 18:59:56.985581 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 18:59:56.985963 930 webserver.cc:460] Webserver started at http://127.0.98.62:43515/ using document root <none> and password file <none>
I20260925 18:59:56.986164 930 fs_manager.cc:359] Metadata directory not provided
I20260925 18:59:56.986205 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:59:56.986361 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:59:56.987313 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "702e29c45a884f78a2bf43fe45de769d"
format_stamp: "Formatted at 2026-09-25 18:59:56 on dist-test-slave-49bz"
server_key: "3cbe5b9c2c76b5f3cc0d169a212cf4ab"
server_key_iv: "103b013d74d86212d1c71db3a8456258"
server_key_version: "encryptionkey@0"
I20260925 18:59:56.987641 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "702e29c45a884f78a2bf43fe45de769d"
format_stamp: "Formatted at 2026-09-25 18:59:56 on dist-test-slave-49bz"
server_key: "3cbe5b9c2c76b5f3cc0d169a212cf4ab"
server_key_iv: "103b013d74d86212d1c71db3a8456258"
server_key_version: "encryptionkey@0"
I20260925 18:59:56.988884 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 18:59:56.989540 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:56.989724 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 18:59:56.989830 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "702e29c45a884f78a2bf43fe45de769d"
format_stamp: "Formatted at 2026-09-25 18:59:56 on dist-test-slave-49bz"
server_key: "3cbe5b9c2c76b5f3cc0d169a212cf4ab"
server_key_iv: "103b013d74d86212d1c71db3a8456258"
server_key_version: "encryptionkey@0"
I20260925 18:59:56.989897 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:59:57.006561 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:59:57.008298 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 18:59:57.008435 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:59:57.012358 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37579
I20260925 18:59:57.012409 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37579 every 8 connection(s)
I20260925 18:59:57.012732 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 18:59:57.012874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 930
I20260925 18:59:57.012969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 18:59:57.013151 392 external_mini_cluster.cc:1428] Setting key 169471b6065c9fd9e6273cb00b06de81
I20260925 18:59:57.013710 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:59:57.016536 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d: Bootstrap starting.
I20260925 18:59:57.017196 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d: Neither blocks nor log segments found. Creating new log.
I20260925 18:59:57.017474 998 log.cc:824] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d: Log is configured to *not* fsync() on all Append() calls
I20260925 18:59:57.018093 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d: No bootstrap required, opened a new log
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 18:59:57.019953 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702e29c45a884f78a2bf43fe45de769d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } }
I20260925 18:59:57.020097 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:59:57.020143 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 702e29c45a884f78a2bf43fe45de769d, State: Initialized, Role: FOLLOWER
I20260925 18:59:57.020304 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702e29c45a884f78a2bf43fe45de769d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } }
I20260925 18:59:57.020408 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:59:57.020442 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:59:57.020490 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:59:57.021126 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702e29c45a884f78a2bf43fe45de769d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } }
I20260925 18:59:57.021203 998 leader_election.cc:302] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 702e29c45a884f78a2bf43fe45de769d; no voters:
I20260925 18:59:57.021368 998 leader_election.cc:288] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:59:57.021415 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:59:57.021533 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [term 1 LEADER]: Becoming Leader. State: Replica: 702e29c45a884f78a2bf43fe45de769d, State: Running, Role: LEADER
I20260925 18:59:57.021626 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:59:57.021611 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702e29c45a884f78a2bf43fe45de769d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } }
I20260925 18:59:57.021926 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "702e29c45a884f78a2bf43fe45de769d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702e29c45a884f78a2bf43fe45de769d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } } }
I20260925 18:59:57.022007 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [sys.catalog]: This master's current role is: LEADER
I20260925 18:59:57.022171 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 702e29c45a884f78a2bf43fe45de769d. Latest consensus state: current_term: 1 leader_uuid: "702e29c45a884f78a2bf43fe45de769d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702e29c45a884f78a2bf43fe45de769d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } } }
I20260925 18:59:57.022234 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d [sys.catalog]: This master's current role is: LEADER
I20260925 18:59:57.022570 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:59:57.023202 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:59:57.023809 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.013657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.0.98.62:37579)
0925 18:59:57.013855 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.013864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.013895 (+ 31us) server_negotiation.cc:407] Connection header received
0925 18:59:57.014349 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.014357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.014417 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.014531 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:57.015195 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.016065 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.017062 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.017532 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.017631 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.019974 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:57.019978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:57.019978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:57.019996 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:57.022006 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.022468 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.022471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.022471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.022573 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.022788 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.022789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.022789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.023068 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:57.023203 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.023377 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.023618 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
I20260925 18:59:57.024585 1009 catalog_manager.cc:1347] Generated new cluster ID: d1c2e7315800420f9e431ed43f281bcf
I20260925 18:59:57.024633 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:59:57.039991 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:59:57.040472 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:59:57.042703 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 702e29c45a884f78a2bf43fe45de769d: Generated new TSK 0
I20260925 18:59:57.042879 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 18:59:57.072165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37579
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 18:59:57.153729 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 18:59:57.153928 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 18:59:57.153951 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 18:59:57.155175 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 18:59:57.155216 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 18:59:57.155267 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 18:59:57.156721 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 18:59:57.156956 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 18:59:57.157186 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:59:57.161933 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:57.161935 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:57.161950 1024 server_base.cc:1034] running on GCE node
W20260925 18:59:57.161935 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:57.162408 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 18:59:57.162592 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 18:59:57.163748 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790362797163724 us; error 28 us; skew 500 ppm
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 18:59:57.166091 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 18:59:57.166538 1024 webserver.cc:460] Webserver started at http://127.0.98.1:33131/ using document root <none> and password file <none>
I20260925 18:59:57.166703 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 18:59:57.166749 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:59:57.166831 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:59:57.167758 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "b8e272e4ea2e431fba87c5209ccafd6b"
format_stamp: "Formatted at 2026-09-25 18:59:57 on dist-test-slave-49bz"
server_key: "8374160b2fb99051615ce9a23d28e993"
server_key_iv: "8c46af07b11a78b3c1cae8e03386379c"
server_key_version: "encryptionkey@0"
I20260925 18:59:57.168046 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b8e272e4ea2e431fba87c5209ccafd6b"
format_stamp: "Formatted at 2026-09-25 18:59:57 on dist-test-slave-49bz"
server_key: "8374160b2fb99051615ce9a23d28e993"
server_key_iv: "8c46af07b11a78b3c1cae8e03386379c"
server_key_version: "encryptionkey@0"
I20260925 18:59:57.169247 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 18:59:57.169924 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:57.170112 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 18:59:57.170207 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "b8e272e4ea2e431fba87c5209ccafd6b"
format_stamp: "Formatted at 2026-09-25 18:59:57 on dist-test-slave-49bz"
server_key: "8374160b2fb99051615ce9a23d28e993"
server_key_iv: "8c46af07b11a78b3c1cae8e03386379c"
server_key_version: "encryptionkey@0"
I20260925 18:59:57.170310 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:59:57.181752 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:59:57.183456 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 18:59:57.183593 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:59:57.183808 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:59:57.184082 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:59:57.184110 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:57.184129 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:59:57.184170 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:57.189618 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42357
I20260925 18:59:57.189690 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42357 every 8 connection(s)
I20260925 18:59:57.189960 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 18:59:57.196707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 1024
I20260925 18:59:57.196784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 18:59:57.196935 392 external_mini_cluster.cc:1428] Setting key a95e3c210593ba7b4b76c3881702c3b9
I20260925 18:59:57.198565 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.190615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44469 (local address 127.0.98.62:37579)
0925 18:59:57.190652 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.190653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.190817 (+ 164us) server_negotiation.cc:407] Connection header received
0925 18:59:57.191212 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.191213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.191222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.191272 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:57.192059 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.192536 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.193478 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.193648 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.193690 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.195645 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:57.195646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:57.195646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:57.195653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:57.197184 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.197593 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.197594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.197595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.197656 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.197914 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.197915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.197915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.198024 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:57.198093 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.198397 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.198485 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 18:59:57.198904 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.190573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37579 (local address 127.0.98.1:44469)
0925 18:59:57.190714 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:57.190748 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:57.191142 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:57.191318 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:57.191321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:57.191826 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 18:59:57.191989 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.191993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.192611 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.192612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.193404 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.193406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.193706 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.193707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.193796 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.194231 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 18:59:57.194238 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 18:59:57.195568 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 18:59:57.197267 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:57.197273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:57.197275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:57.197522 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:57.197716 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:57.197717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:57.197717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:57.197836 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:57.198099 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 18:59:57.198105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 18:59:57.198228 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 18:59:57.198377 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 18:59:57.198522 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260925 18:59:57.199385 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37579
I20260925 18:59:57.199455 1153 heartbeater.cc:464] Registering TS with master...
I20260925 18:59:57.199627 1153 heartbeater.cc:511] Master 127.0.98.62:37579 requested a full tablet report, sending...
I20260925 18:59:57.200098 962 ts_manager.cc:194] Registered new tserver with Master: b8e272e4ea2e431fba87c5209ccafd6b (127.0.98.1:42357)
I20260925 18:59:57.200702 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44469
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 18:59:57.222183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37579
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 18:59:57.298373 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 18:59:57.298548 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 18:59:57.298571 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 18:59:57.299652 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 18:59:57.299685 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 18:59:57.299710 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 18:59:57.300980 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 18:59:57.301151 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 18:59:57.301292 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:59:57.305757 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:57.305739 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:57.305729 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:57.306025 1158 server_base.cc:1034] running on GCE node
I20260925 18:59:57.306190 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 18:59:57.306425 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 18:59:57.307557 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790362797307547 us; error 27 us; skew 500 ppm
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 18:59:57.310051 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 18:59:57.310491 1158 webserver.cc:460] Webserver started at http://127.0.98.2:38863/ using document root <none> and password file <none>
I20260925 18:59:57.310685 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 18:59:57.310724 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:59:57.310854 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:59:57.311801 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "e8726b3d91224f8094f5a8f61c9d4953"
format_stamp: "Formatted at 2026-09-25 18:59:57 on dist-test-slave-49bz"
server_key: "0684b1b672ea665f4d0c7dbc84f5cf43"
server_key_iv: "74cf03332bb0e950d1b44273612340e4"
server_key_version: "encryptionkey@0"
I20260925 18:59:57.312129 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e8726b3d91224f8094f5a8f61c9d4953"
format_stamp: "Formatted at 2026-09-25 18:59:57 on dist-test-slave-49bz"
server_key: "0684b1b672ea665f4d0c7dbc84f5cf43"
server_key_iv: "74cf03332bb0e950d1b44273612340e4"
server_key_version: "encryptionkey@0"
I20260925 18:59:57.313375 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 18:59:57.314054 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:57.314226 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 18:59:57.314385 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "e8726b3d91224f8094f5a8f61c9d4953"
format_stamp: "Formatted at 2026-09-25 18:59:57 on dist-test-slave-49bz"
server_key: "0684b1b672ea665f4d0c7dbc84f5cf43"
server_key_iv: "74cf03332bb0e950d1b44273612340e4"
server_key_version: "encryptionkey@0"
I20260925 18:59:57.314484 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:59:57.335855 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:59:57.337502 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 18:59:57.337652 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:59:57.337862 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:59:57.338145 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:59:57.338195 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:57.338229 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:59:57.338289 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:57.343647 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44463
I20260925 18:59:57.343693 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44463 every 8 connection(s)
I20260925 18:59:57.343976 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 18:59:57.346525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 1158
I20260925 18:59:57.346621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 18:59:57.346779 392 external_mini_cluster.cc:1428] Setting key 2cae9b9c58c04c7567265796aedfe569
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 18:59:57.353148 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.344551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37579 (local address 127.0.98.2:41275)
0925 18:59:57.344690 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:57.344725 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:57.345135 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:57.345312 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:57.345315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:57.345788 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 18:59:57.345948 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.345952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.346607 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.346608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.347821 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.347824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.348158 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.348159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.348328 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.348768 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 18:59:57.348777 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 18:59:57.350335 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 18:59:57.351986 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:57.351991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:57.351992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:57.352187 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:57.352382 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:57.352382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:57.352382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:57.352463 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:57.352721 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 18:59:57.352724 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 18:59:57.352833 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 18:59:57.352903 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 18:59:57.352998 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260925 18:59:57.353148 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.344574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41275 (local address 127.0.98.62:37579)
0925 18:59:57.344634 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.344636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.344818 (+ 182us) server_negotiation.cc:407] Connection header received
0925 18:59:57.345203 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.345204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.345212 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.345263 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:57.346014 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.346526 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.347903 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.348083 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.348136 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.350410 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:57.350411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:57.350411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:57.350416 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:57.351915 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.352258 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.352259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.352260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.352319 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.352526 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.352527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.352527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.352662 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:57.352728 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.352947 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.353032 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 18:59:57.353595 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37579
I20260925 18:59:57.353699 1287 heartbeater.cc:464] Registering TS with master...
I20260925 18:59:57.353869 1287 heartbeater.cc:511] Master 127.0.98.62:37579 requested a full tablet report, sending...
I20260925 18:59:57.354208 962 ts_manager.cc:194] Registered new tserver with Master: e8726b3d91224f8094f5a8f61c9d4953 (127.0.98.2:44463)
I20260925 18:59:57.354728 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41275
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 18:59:57.371047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37579
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 18:59:57.447036 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 18:59:57.447228 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 18:59:57.447250 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 18:59:57.448386 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 18:59:57.448424 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 18:59:57.448473 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 18:59:57.449802 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 18:59:57.450012 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 18:59:57.450204 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:59:57.454692 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:57.454684 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:59:57.454692 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:59:57.454867 1292 server_base.cc:1034] running on GCE node
I20260925 18:59:57.455101 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 18:59:57.455384 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 18:59:57.456540 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790362797456521 us; error 37 us; skew 500 ppm
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 18:59:57.458812 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 18:59:57.459218 1292 webserver.cc:460] Webserver started at http://127.0.98.3:37641/ using document root <none> and password file <none>
I20260925 18:59:57.459460 1292 fs_manager.cc:359] Metadata directory not provided
I20260925 18:59:57.459524 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:59:57.459666 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:59:57.460608 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "6cda2faa069f44f887fd4a82a7ca4bc9"
format_stamp: "Formatted at 2026-09-25 18:59:57 on dist-test-slave-49bz"
server_key: "b259f423cc8d0321265c2d4d63a778be"
server_key_iv: "a6358642ad3c0e64ba3c580909b2f13a"
server_key_version: "encryptionkey@0"
I20260925 18:59:57.460945 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6cda2faa069f44f887fd4a82a7ca4bc9"
format_stamp: "Formatted at 2026-09-25 18:59:57 on dist-test-slave-49bz"
server_key: "b259f423cc8d0321265c2d4d63a778be"
server_key_iv: "a6358642ad3c0e64ba3c580909b2f13a"
server_key_version: "encryptionkey@0"
I20260925 18:59:57.462111 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 18:59:57.462831 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:57.462994 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:57.463081 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "6cda2faa069f44f887fd4a82a7ca4bc9"
format_stamp: "Formatted at 2026-09-25 18:59:57 on dist-test-slave-49bz"
server_key: "b259f423cc8d0321265c2d4d63a778be"
server_key_iv: "a6358642ad3c0e64ba3c580909b2f13a"
server_key_version: "encryptionkey@0"
I20260925 18:59:57.463161 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:59:57.474967 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:59:57.476531 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 18:59:57.476668 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:59:57.476861 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:59:57.477131 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:59:57.477186 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:57.477223 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:59:57.477252 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:59:57.482785 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46619
I20260925 18:59:57.482832 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46619 every 8 connection(s)
I20260925 18:59:57.483119 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 18:59:57.484910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 1292
I20260925 18:59:57.485025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 18:59:57.485185 392 external_mini_cluster.cc:1428] Setting key 9873de09e6a7290b0c760767498d5294
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 18:59:57.491632 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.483861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53295 (local address 127.0.98.62:37579)
0925 18:59:57.483918 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.483920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.484073 (+ 153us) server_negotiation.cc:407] Connection header received
0925 18:59:57.484526 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.484527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.484540 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.484584 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:57.485709 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.486385 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.487292 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.487463 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.487511 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.489383 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:57.489385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:57.489385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:57.489403 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:57.490588 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.490940 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.490941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.490941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.490988 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.491155 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.491155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.491155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.491243 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:57.491300 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.491508 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.491562 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 18:59:57.491773 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.483830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37579 (local address 127.0.98.3:53295)
0925 18:59:57.483970 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:57.484000 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:57.484425 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:57.484659 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:57.484662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:57.485385 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0925 18:59:57.485615 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.485619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.486456 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.486457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.487218 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.487220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.487528 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.487528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.487616 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.487984 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 18:59:57.487991 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 18:59:57.489307 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 18:59:57.490653 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:57.490658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:57.490659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:57.490874 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:57.491029 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 18:59:57.491029 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 18:59:57.491029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 18:59:57.491105 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 18:59:57.491308 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 18:59:57.491312 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 18:59:57.491407 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 18:59:57.491483 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 18:59:57.491584 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260925 18:59:57.492211 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37579
I20260925 18:59:57.492303 1421 heartbeater.cc:464] Registering TS with master...
I20260925 18:59:57.492455 1421 heartbeater.cc:511] Master 127.0.98.62:37579 requested a full tablet report, sending...
I20260925 18:59:57.492816 962 ts_manager.cc:194] Registered new tserver with Master: 6cda2faa069f44f887fd4a82a7ca4bc9 (127.0.98.3:46619)
I20260925 18:59:57.493184 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53295
I20260925 18:59:57.496812 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 18:59:57.503633 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.498304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.0.98.62:37579)
0925 18:59:57.498338 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.498340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.498469 (+ 129us) server_negotiation.cc:407] Connection header received
0925 18:59:57.498492 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.498493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.498500 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.498539 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:57.499022 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.499433 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.500091 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.500246 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.500296 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.501330 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:57.501331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:57.501331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:57.501335 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:57.502473 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.502800 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.502801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.502801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.502852 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.503078 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.503078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.503079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.503248 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:57.503352 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.503480 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.503544 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 18:59:57.505443 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 18:59:57.506975 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 18:59:57.513339 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.508836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.0.98.3:46619)
0925 18:59:57.509592 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.509596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.509605 (+ 9us) server_negotiation.cc:407] Connection header received
0925 18:59:57.509652 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.509654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.509669 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.509746 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 18:59:57.510499 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.511352 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.512514 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.512963 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.513042 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.513102 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.513235 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.513269 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":82,"threads_started":1}
I20260925 18:59:57.513370 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.508772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46619 (local address 127.0.0.1:58426)
0925 18:59:57.508891 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:57.508925 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:57.508994 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:57.510373 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:57.510373 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:57.510375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 18:59:57.510418 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.510420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.511453 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.511454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.512443 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.512444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.513065 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.513066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.513138 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.513148 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 18:59:57.513224 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 18:59:57.513275 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":79,"threads_started":1}
I20260925 18:59:57.513430 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.509225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.0.98.1:42357)
0925 18:59:57.509366 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.509369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.509510 (+ 141us) server_negotiation.cc:407] Connection header received
0925 18:59:57.509548 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.509550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.509564 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.509652 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 18:59:57.509868 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.511003 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.512315 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.512972 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.513028 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.513083 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.513315 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.513355 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260925 18:59:57.513597 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.508994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44463 (local address 127.0.0.1:50724)
0925 18:59:57.509325 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:57.509334 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:57.509389 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:57.509595 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:57.509597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:57.509600 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 18:59:57.509677 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.509678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.511028 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.511029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.512116 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.512118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.513086 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.513088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.513149 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.513153 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 18:59:57.513228 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 18:59:57.513261 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":31,"thread_start_us":44,"threads_started":1}
I20260925 18:59:57.513365 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.509121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42357 (local address 127.0.0.1:34874)
0925 18:59:57.509453 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:57.509459 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:57.509490 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:57.509730 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:57.509731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:57.509733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 18:59:57.509784 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.509785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.511085 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.511085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.512231 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.512233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.513107 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.513107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.513153 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.513156 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 18:59:57.513201 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 18:59:57.513254 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":36,"thread_start_us":62,"threads_started":1}
I20260925 18:59:57.513684 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.509077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.0.98.2:44463)
0925 18:59:57.509223 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.509226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.509405 (+ 179us) server_negotiation.cc:407] Connection header received
0925 18:59:57.509447 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.509449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.509469 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.509571 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 18:59:57.509751 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.510930 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.512204 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.512914 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.512986 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.513043 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.513567 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.513606 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260925 18:59:57.514230 1221 tablet_service.cc:1467] Processing CreateTablet for tablet d540675238514f4d9f5d64ff57d72939 (DEFAULT_TABLE table=test-table [id=d5532d4701fb46259d0206aa7f59a383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:59:57.514325 1355 tablet_service.cc:1467] Processing CreateTablet for tablet d540675238514f4d9f5d64ff57d72939 (DEFAULT_TABLE table=test-table [id=d5532d4701fb46259d0206aa7f59a383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:59:57.514335 1087 tablet_service.cc:1467] Processing CreateTablet for tablet d540675238514f4d9f5d64ff57d72939 (DEFAULT_TABLE table=test-table [id=d5532d4701fb46259d0206aa7f59a383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:59:57.514596 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d540675238514f4d9f5d64ff57d72939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:59:57.514612 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d540675238514f4d9f5d64ff57d72939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:59:57.514613 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d540675238514f4d9f5d64ff57d72939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:59:57.516880 1435 tablet_bootstrap.cc:492] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953: Bootstrap starting.
I20260925 18:59:57.517417 1435 tablet_bootstrap.cc:654] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953: Neither blocks nor log segments found. Creating new log.
I20260925 18:59:57.517689 1436 tablet_bootstrap.cc:492] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9: Bootstrap starting.
I20260925 18:59:57.518074 1437 tablet_bootstrap.cc:492] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b: Bootstrap starting.
I20260925 18:59:57.518082 1435 log.cc:824] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953: Log is configured to *not* fsync() on all Append() calls
I20260925 18:59:57.518621 1436 tablet_bootstrap.cc:654] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9: Neither blocks nor log segments found. Creating new log.
I20260925 18:59:57.519192 1436 log.cc:824] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9: Log is configured to *not* fsync() on all Append() calls
I20260925 18:59:57.519223 1437 tablet_bootstrap.cc:654] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b: Neither blocks nor log segments found. Creating new log.
I20260925 18:59:57.519372 1435 tablet_bootstrap.cc:492] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953: No bootstrap required, opened a new log
I20260925 18:59:57.519443 1435 ts_tablet_manager.cc:1397] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 18:59:57.519718 1437 log.cc:824] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b: Log is configured to *not* fsync() on all Append() calls
I20260925 18:59:57.520000 1436 tablet_bootstrap.cc:492] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9: No bootstrap required, opened a new log
I20260925 18:59:57.520063 1436 ts_tablet_manager.cc:1397] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 18:59:57.520398 1437 tablet_bootstrap.cc:492] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b: No bootstrap required, opened a new log
I20260925 18:59:57.520503 1437 ts_tablet_manager.cc:1397] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 18:59:57.521303 1435 raft_consensus.cc:348] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46619 } } peers { permanent_uuid: "e8726b3d91224f8094f5a8f61c9d4953" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44463 } } peers { permanent_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42357 } }
I20260925 18:59:57.521457 1435 raft_consensus.cc:374] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:59:57.521497 1435 raft_consensus.cc:729] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8726b3d91224f8094f5a8f61c9d4953, State: Initialized, Role: FOLLOWER
I20260925 18:59:57.521595 1435 consensus_queue.cc:260] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46619 } } peers { permanent_uuid: "e8726b3d91224f8094f5a8f61c9d4953" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44463 } } peers { permanent_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42357 } }
I20260925 18:59:57.521828 1435 ts_tablet_manager.cc:1428] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 18:59:57.521934 1287 heartbeater.cc:503] Master 127.0.98.62:37579 was elected leader, sending a full tablet report...
I20260925 18:59:57.521822 1436 raft_consensus.cc:348] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46619 } } peers { permanent_uuid: "e8726b3d91224f8094f5a8f61c9d4953" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44463 } } peers { permanent_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42357 } }
I20260925 18:59:57.521966 1436 raft_consensus.cc:374] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:59:57.522040 1436 raft_consensus.cc:729] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cda2faa069f44f887fd4a82a7ca4bc9, State: Initialized, Role: FOLLOWER
I20260925 18:59:57.522140 1436 consensus_queue.cc:260] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46619 } } peers { permanent_uuid: "e8726b3d91224f8094f5a8f61c9d4953" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44463 } } peers { permanent_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42357 } }
I20260925 18:59:57.522197 1437 raft_consensus.cc:348] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46619 } } peers { permanent_uuid: "e8726b3d91224f8094f5a8f61c9d4953" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44463 } } peers { permanent_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42357 } }
I20260925 18:59:57.522365 1437 raft_consensus.cc:374] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:59:57.522398 1437 raft_consensus.cc:729] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e272e4ea2e431fba87c5209ccafd6b, State: Initialized, Role: FOLLOWER
I20260925 18:59:57.522399 1421 heartbeater.cc:503] Master 127.0.98.62:37579 was elected leader, sending a full tablet report...
I20260925 18:59:57.522424 1436 ts_tablet_manager.cc:1428] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 18:59:57.522522 1437 consensus_queue.cc:260] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46619 } } peers { permanent_uuid: "e8726b3d91224f8094f5a8f61c9d4953" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44463 } } peers { permanent_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42357 } }
I20260925 18:59:57.522755 1437 ts_tablet_manager.cc:1428] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 18:59:57.522881 1153 heartbeater.cc:503] Master 127.0.98.62:37579 was elected leader, sending a full tablet report...
W20260925 18:59:57.594483 1288 tablet.cc:2369] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 18:59:57.690623 1154 tablet.cc:2369] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 18:59:57.733626 1422 tablet.cc:2369] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 18:59:57.762276 1443 raft_consensus.cc:482] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:59:57.762424 1443 raft_consensus.cc:504] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46619 } } peers { permanent_uuid: "e8726b3d91224f8094f5a8f61c9d4953" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44463 } } peers { permanent_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42357 } }
I20260925 18:59:57.762761 1443 leader_election.cc:288] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cda2faa069f44f887fd4a82a7ca4bc9 (127.0.98.3:46619), e8726b3d91224f8094f5a8f61c9d4953 (127.0.98.2:44463)
I20260925 18:59:57.766683 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.763049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36757 (local address 127.0.98.3:46619)
0925 18:59:57.763121 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.763124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.763188 (+ 64us) server_negotiation.cc:407] Connection header received
0925 18:59:57.763300 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.763302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.763314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.763410 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 18:59:57.763594 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.764274 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.765702 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.766322 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.766380 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.766388 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.766560 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.766616 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 18:59:57.766683 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.763230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60585 (local address 127.0.98.2:44463)
0925 18:59:57.763271 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.763273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.763296 (+ 23us) server_negotiation.cc:407] Connection header received
0925 18:59:57.763380 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.763381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.763392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.763452 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 18:59:57.763594 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.764268 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.765703 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.766299 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.766372 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.766383 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.766560 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.766616 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 18:59:57.766747 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.763169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44463 (local address 127.0.98.1:60585)
0925 18:59:57.763247 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:57.763256 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:57.763317 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:57.763461 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:57.763461 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:57.763475 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 18:59:57.763533 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.763534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.764345 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.764346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.765617 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.765619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.766391 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.766391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.766474 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.766480 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 18:59:57.766547 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 18:59:57.766617 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":63,"thread_start_us":39,"threads_started":1}
I20260925 18:59:57.766964 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.762962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46619 (local address 127.0.98.1:36757)
0925 18:59:57.763112 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 18:59:57.763126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 18:59:57.763206 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 18:59:57.763423 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 18:59:57.763424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 18:59:57.763428 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 18:59:57.763513 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.763515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.764348 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.764348 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.765616 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 18:59:57.765619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.766390 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.766391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 18:59:57.766474 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.766481 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 18:59:57.766547 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 18:59:57.766617 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":60,"thread_start_us":75,"threads_started":1}
I20260925 18:59:57.767136 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d540675238514f4d9f5d64ff57d72939" candidate_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8726b3d91224f8094f5a8f61c9d4953" is_pre_election: true
I20260925 18:59:57.767297 1241 raft_consensus.cc:2454] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e272e4ea2e431fba87c5209ccafd6b in term 0.
I20260925 18:59:57.767539 1041 leader_election.cc:302] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8e272e4ea2e431fba87c5209ccafd6b, e8726b3d91224f8094f5a8f61c9d4953; no voters:
I20260925 18:59:57.767670 1443 raft_consensus.cc:2781] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 18:59:57.767668 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d540675238514f4d9f5d64ff57d72939" candidate_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" is_pre_election: true
I20260925 18:59:57.767724 1443 raft_consensus.cc:482] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:59:57.767781 1443 raft_consensus.cc:3037] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:59:57.767822 1375 raft_consensus.cc:2454] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e272e4ea2e431fba87c5209ccafd6b in term 0.
I20260925 18:59:57.824397 1443 raft_consensus.cc:504] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46619 } } peers { permanent_uuid: "e8726b3d91224f8094f5a8f61c9d4953" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44463 } } peers { permanent_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42357 } }
I20260925 18:59:57.824662 1443 leader_election.cc:288] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [CANDIDATE]: Term 1 election: Requested vote from peers 6cda2faa069f44f887fd4a82a7ca4bc9 (127.0.98.3:46619), e8726b3d91224f8094f5a8f61c9d4953 (127.0.98.2:44463)
I20260925 18:59:57.824936 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d540675238514f4d9f5d64ff57d72939" candidate_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9"
I20260925 18:59:57.824934 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d540675238514f4d9f5d64ff57d72939" candidate_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8726b3d91224f8094f5a8f61c9d4953"
I20260925 18:59:57.825045 1241 raft_consensus.cc:3037] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:59:57.825040 1375 raft_consensus.cc:3037] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:59:57.825800 1375 raft_consensus.cc:2454] T d540675238514f4d9f5d64ff57d72939 P 6cda2faa069f44f887fd4a82a7ca4bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e272e4ea2e431fba87c5209ccafd6b in term 1.
I20260925 18:59:57.825985 1041 leader_election.cc:302] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cda2faa069f44f887fd4a82a7ca4bc9, b8e272e4ea2e431fba87c5209ccafd6b; no voters:
I20260925 18:59:57.826102 1443 raft_consensus.cc:2781] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:59:57.826284 1443 raft_consensus.cc:686] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [term 1 LEADER]: Becoming Leader. State: Replica: b8e272e4ea2e431fba87c5209ccafd6b, State: Running, Role: LEADER
I20260925 18:59:57.826506 1443 consensus_queue.cc:237] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46619 } } peers { permanent_uuid: "e8726b3d91224f8094f5a8f61c9d4953" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44463 } } peers { permanent_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42357 } }
I20260925 18:59:57.826586 1241 raft_consensus.cc:2454] T d540675238514f4d9f5d64ff57d72939 P e8726b3d91224f8094f5a8f61c9d4953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e272e4ea2e431fba87c5209ccafd6b in term 1.
I20260925 18:59:57.827591 962 catalog_manager.cc:5495] T d540675238514f4d9f5d64ff57d72939 P b8e272e4ea2e431fba87c5209ccafd6b reported cstate change: term changed from 0 to 1, leader changed from <none> to b8e272e4ea2e431fba87c5209ccafd6b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cda2faa069f44f887fd4a82a7ca4bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8726b3d91224f8094f5a8f61c9d4953" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8e272e4ea2e431fba87c5209ccafd6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42357 } health_report { overall_health: HEALTHY } } }
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 18:59:57.871598 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.864705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.0.98.1:42357)
0925 18:59:57.864792 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.864794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.864840 (+ 46us) server_negotiation.cc:407] Connection header received
0925 18:59:57.864883 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.864884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.864895 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.864964 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:57.865504 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.865962 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.866739 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.866930 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.867024 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.869176 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:57.869185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:57.869185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:57.869191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:57.870664 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.870974 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.870977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.870977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.871038 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.871182 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.871183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.871183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.871298 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:57.871395 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.871444 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.871496 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 18:59:57.930918 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:57.922231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.0.98.1:42357)
0925 18:59:57.922292 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:57.922295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:57.923619 (+ 1324us) server_negotiation.cc:407] Connection header received
0925 18:59:57.923721 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:57.923722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:57.923732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:57.923779 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:57.924364 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.924965 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.925718 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:57.925954 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:57.926042 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:57.928408 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:57.928409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:57.928409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:57.928417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:57.929702 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.930035 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.930037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.930037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.930094 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:57.930337 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:57.930338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:57.930339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:57.930435 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:57.930533 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:57.930730 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 18:59:57.930813 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 18:59:57 dist-test-slave-49bz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 18:59:58 dist-test-slave-49bz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 18:59:58.029500 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 18:59:58.020091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.0.98.1:42357)
0925 18:59:58.020186 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 18:59:58.020188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 18:59:58.020425 (+ 237us) server_negotiation.cc:407] Connection header received
0925 18:59:58.020500 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 18:59:58.020501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 18:59:58.020511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 18:59:58.020569 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 18:59:58.021249 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:58.022059 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:58.023113 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 18:59:58.023398 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 18:59:58.023481 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 18:59:58.026626 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 18:59:58.026627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 18:59:58.026628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 18:59:58.026634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 18:59:58.028432 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:58.028740 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:58.028741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:58.028741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:58.028817 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 18:59:58.028987 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 18:59:58.028988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 18:59:58.028988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 18:59:58.029108 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 18:59:58.029219 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 18:59:58.029306 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 18:59:58.029393 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 18:59:58.030002 1067 tablet_service.cc:2908] Rejecting scan request for tablet d540675238514f4d9f5d64ff57d72939: Uninitialized: clean time has not yet been initialized
I20260925 18:59:58.030721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 1024
I20260925 18:59:58.035712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 1158
I20260925 18:59:58.040318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 1292
I20260925 18:59:58.044667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 930
2026-09-25T18:59:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2255 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:00 dist-test-slave-49bz krb5kdc[1460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:00 dist-test-slave-49bz krb5kdc[1460](info): set up 2 sockets
Sep 25 19:00:00 dist-test-slave-49bz krb5kdc[1460](info): commencing operation
krb5kdc: starting...
W20260925 19:00:01.065263 392 mini_kdc.cc:121] Time spent starting KDC: real 3.000s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 19:00:01.245486 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.180s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 19:00:01.381825 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.136s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 19:00:01.501178 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.119s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/test-user.keytab.
W20260925 19:00:01.973843 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.473s user 0.000s sys 0.001s
Sep 25 19:00:01 dist-test-slave-49bz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260925 19:00:02.716507 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.723s user 0.000s sys 0.002s
I20260925 19:00:02.716817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37745
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:02.796752 1473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:02.796944 1473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:02.796967 1473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:02.798058 1473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:02.798095 1473 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 19:00:02.798110 1473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:02.798122 1473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:02.798134 1473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:02.798146 1473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:02.799602 1473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37745
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:02.799779 1473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:02.799942 1473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:00:02.804708 1473 server_base.cc:1034] running on GCE node
W20260925 19:00:02.804718 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:02.804690 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:02.804687 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:02.805236 1473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:02.805534 1473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:02.806687 1473 hybrid_clock.cc:648] HybridClock initialized: now 1790362802806682 us; error 38 us; skew 500 ppm
Sep 25 19:00:02 dist-test-slave-49bz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:02.809129 1473 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:00:02.809576 1473 webserver.cc:460] Webserver started at http://127.0.98.62:33039/ using document root <none> and password file <none>
I20260925 19:00:02.809834 1473 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:02.809903 1473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:02.810045 1473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:02.811136 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "1445a9468a694150bcec064903d45c36"
format_stamp: "Formatted at 2026-09-25 19:00:02 on dist-test-slave-49bz"
server_key: "7dcbdd68fa0ff5972312612a64326eea"
server_key_iv: "044a359d15a68524a5e6f54b4d979ee0"
server_key_version: "encryptionkey@0"
I20260925 19:00:02.811503 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "1445a9468a694150bcec064903d45c36"
format_stamp: "Formatted at 2026-09-25 19:00:02 on dist-test-slave-49bz"
server_key: "7dcbdd68fa0ff5972312612a64326eea"
server_key_iv: "044a359d15a68524a5e6f54b4d979ee0"
server_key_version: "encryptionkey@0"
I20260925 19:00:02.812806 1473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:00:02.813548 1488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:02.813692 1473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:02.813792 1473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "1445a9468a694150bcec064903d45c36"
format_stamp: "Formatted at 2026-09-25 19:00:02 on dist-test-slave-49bz"
server_key: "7dcbdd68fa0ff5972312612a64326eea"
server_key_iv: "044a359d15a68524a5e6f54b4d979ee0"
server_key_version: "encryptionkey@0"
I20260925 19:00:02.813875 1473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:02.823576 1473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:02.825820 1473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:02.825981 1473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:02.830401 1473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37745
I20260925 19:00:02.830476 1550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37745 every 8 connection(s)
I20260925 19:00:02.830832 1473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:02.831300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 1473
I20260925 19:00:02.831405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:02.831578 392 external_mini_cluster.cc:1428] Setting key 57e1f742d025dfbd09384b004e1844c0
I20260925 19:00:02.831612 1551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:02.834406 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36: Bootstrap starting.
I20260925 19:00:02.835018 1551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:02.835345 1551 log.cc:824] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:02.836056 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36: No bootstrap required, opened a new log
Sep 25 19:00:02 dist-test-slave-49bz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:02.838239 1551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1445a9468a694150bcec064903d45c36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37745 } }
I20260925 19:00:02.838426 1551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:02.838474 1551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1445a9468a694150bcec064903d45c36, State: Initialized, Role: FOLLOWER
I20260925 19:00:02.838617 1551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1445a9468a694150bcec064903d45c36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37745 } }
I20260925 19:00:02.838678 1551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:02.838698 1551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:02.838764 1551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:02.841775 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:02.832068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.0.98.62:37745)
0925 19:00:02.832250 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:02.832260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:02.832292 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:00:02.832738 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:02.832745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:02.832792 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:02.832941 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:02.833553 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:02.834598 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:02.835429 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:02.835826 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:02.835965 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:02.838447 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:02.838452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:02.838452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:02.838463 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:02.840246 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:02.840634 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:02.840637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:02.840637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:02.840749 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:02.840990 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:02.840991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:02.840992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:02.841217 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:02.841370 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:02.841466 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 19:00:02.841604 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260925 19:00:02.882539 1551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1445a9468a694150bcec064903d45c36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37745 } }
I20260925 19:00:02.882736 1551 leader_election.cc:302] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1445a9468a694150bcec064903d45c36; no voters:
I20260925 19:00:02.882956 1551 leader_election.cc:288] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:02.883028 1556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:02.883239 1551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:02.883253 1556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [term 1 LEADER]: Becoming Leader. State: Replica: 1445a9468a694150bcec064903d45c36, State: Running, Role: LEADER
I20260925 19:00:02.883508 1556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1445a9468a694150bcec064903d45c36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37745 } }
I20260925 19:00:02.884131 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1445a9468a694150bcec064903d45c36. Latest consensus state: current_term: 1 leader_uuid: "1445a9468a694150bcec064903d45c36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1445a9468a694150bcec064903d45c36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37745 } } }
I20260925 19:00:02.884227 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:02.884378 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1445a9468a694150bcec064903d45c36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1445a9468a694150bcec064903d45c36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37745 } } }
I20260925 19:00:02.884436 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:02.884683 1565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:02.885267 1565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:02.888077 1565 catalog_manager.cc:1347] Generated new cluster ID: b364b4cf35064313adccaff7ee1e548d
I20260925 19:00:02.888149 1565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:02.919379 1565 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:02.919930 1565 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:02.922439 1565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36: Generated new TSK 0
I20260925 19:00:02.922629 1565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260925 19:00:03.400504 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.465s user 0.000s sys 0.003s
I20260925 19:00:03.400835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37745
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:03.476282 1577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:03.476495 1577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:03.476531 1577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:03.477609 1577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:03.477679 1577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:03.477731 1577 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:00:03.477788 1577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:03.479255 1577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:03.479502 1577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:03.479707 1577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:03.484177 1585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:03.484230 1582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:03.484324 1577 server_base.cc:1034] running on GCE node
W20260925 19:00:03.484230 1583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:03.484599 1577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:03.484810 1577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:03.486006 1577 hybrid_clock.cc:648] HybridClock initialized: now 1790362803485993 us; error 28 us; skew 500 ppm
Sep 25 19:00:03 dist-test-slave-49bz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:03.488298 1577 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:00:03.488719 1577 webserver.cc:460] Webserver started at http://127.0.98.1:43039/ using document root <none> and password file <none>
I20260925 19:00:03.488960 1577 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:03.489027 1577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:03.489176 1577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:03.490151 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "9f5eaec249634478a14f05171b04feee"
format_stamp: "Formatted at 2026-09-25 19:00:03 on dist-test-slave-49bz"
server_key: "e360e6769bbd4a777234ad6838f09622"
server_key_iv: "7535adcd951bd5b57753e560350d2302"
server_key_version: "encryptionkey@0"
I20260925 19:00:03.490557 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9f5eaec249634478a14f05171b04feee"
format_stamp: "Formatted at 2026-09-25 19:00:03 on dist-test-slave-49bz"
server_key: "e360e6769bbd4a777234ad6838f09622"
server_key_iv: "7535adcd951bd5b57753e560350d2302"
server_key_version: "encryptionkey@0"
I20260925 19:00:03.491801 1577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:03.492489 1592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.492663 1577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:03.492759 1577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "9f5eaec249634478a14f05171b04feee"
format_stamp: "Formatted at 2026-09-25 19:00:03 on dist-test-slave-49bz"
server_key: "e360e6769bbd4a777234ad6838f09622"
server_key_iv: "7535adcd951bd5b57753e560350d2302"
server_key_version: "encryptionkey@0"
I20260925 19:00:03.492838 1577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:03.503103 1577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:03.504914 1577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:03.505110 1577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:03.505544 1577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:03.505605 1577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.505645 1577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:03.505687 1577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.511986 1577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34363
I20260925 19:00:03.512061 1708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34363 every 8 connection(s)
I20260925 19:00:03.512308 1577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:00:03 dist-test-slave-49bz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:03.514871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 1577
I20260925 19:00:03.514940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:03.515090 392 external_mini_cluster.cc:1428] Setting key c94acc5cb197605d581e874212dabc08
I20260925 19:00:03.515491 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.506061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40947 (local address 127.0.98.62:37745)
0925 19:00:03.506156 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.506158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.506402 (+ 244us) server_negotiation.cc:407] Connection header received
0925 19:00:03.506899 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.506902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.506918 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.507010 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:03.507832 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.508687 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.510139 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.510431 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.510515 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.512916 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:03.512917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:03.512918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:03.512926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:03.514160 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.514558 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.514559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.514560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.514625 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.514862 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.514862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.514863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.514993 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:03.515046 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.515322 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.515403 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
I20260925 19:00:03.515628 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.506000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37745 (local address 127.0.98.1:40947)
0925 19:00:03.506243 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.506312 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.506765 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.507082 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.507085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.507423 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:03.507681 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.507687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.508781 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.508782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.510051 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.510054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.510524 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.510525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.510661 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.511175 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:03.511184 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:03.512805 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:03.514240 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.514246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.514248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.514476 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.514685 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.514685 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.514686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.514788 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.515046 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:03.515052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:03.515184 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.515289 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.515429 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":70,"threads_started":1}
I20260925 19:00:03.516078 1709 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37745
I20260925 19:00:03.516184 1709 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:03.516381 1709 heartbeater.cc:511] Master 127.0.98.62:37745 requested a full tablet report, sending...
I20260925 19:00:03.516933 1504 ts_manager.cc:194] Registered new tserver with Master: 9f5eaec249634478a14f05171b04feee (127.0.98.1:34363)
I20260925 19:00:03.517567 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40947
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 19:00:03.523432 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.517963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34711 (local address 127.0.98.62:37745)
0925 19:00:03.517998 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.518000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.518106 (+ 106us) server_negotiation.cc:407] Connection header received
0925 19:00:03.518192 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.518193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.518202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.518280 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:03.518659 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.519115 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.519772 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.519984 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.520067 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.521036 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:03.521037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:03.521038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:03.521044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:03.522404 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.522731 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.522732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.522733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.522780 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.522991 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.522991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.522992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.523097 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:03.523150 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.523289 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.523353 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:03.523458 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.517920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37745 (local address 127.0.98.1:34711)
0925 19:00:03.518034 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.518045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.518103 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.518289 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.518289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.518502 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:03.518575 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.518576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.519212 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.519213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.519693 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.519695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.520070 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.520071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.520137 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.520647 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:03.520651 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:03.520923 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:03.522493 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.522494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.522494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.522652 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.522857 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.522858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.522858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.522927 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.523144 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:03.523145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:03.523190 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.523269 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.523354 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:00:03.539191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37745
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:03.614730 1718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:03.614907 1718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:03.614930 1718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:03.616101 1718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:03.616139 1718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:03.616158 1718 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:00:03.616209 1718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:03.617707 1718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:03.617947 1718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:03.618165 1718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:03.622669 1726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:03.622684 1724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:03.622769 1718 server_base.cc:1034] running on GCE node
W20260925 19:00:03.622679 1723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:03.623106 1718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:03.623286 1718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:03.624410 1718 hybrid_clock.cc:648] HybridClock initialized: now 1790362803624398 us; error 27 us; skew 500 ppm
Sep 25 19:00:03 dist-test-slave-49bz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:03.626639 1718 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:00:03.627038 1718 webserver.cc:460] Webserver started at http://127.0.98.2:45053/ using document root <none> and password file <none>
I20260925 19:00:03.627251 1718 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:03.627312 1718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:03.627451 1718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:03.628425 1718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "307963cba9764143a83cd3d5d409eb19"
format_stamp: "Formatted at 2026-09-25 19:00:03 on dist-test-slave-49bz"
server_key: "ca02b08aa5ee1ff700a4ed3911c0d9cb"
server_key_iv: "f04af0c358d3a4f6c039c2e59dc8cf8a"
server_key_version: "encryptionkey@0"
I20260925 19:00:03.628760 1718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "307963cba9764143a83cd3d5d409eb19"
format_stamp: "Formatted at 2026-09-25 19:00:03 on dist-test-slave-49bz"
server_key: "ca02b08aa5ee1ff700a4ed3911c0d9cb"
server_key_iv: "f04af0c358d3a4f6c039c2e59dc8cf8a"
server_key_version: "encryptionkey@0"
I20260925 19:00:03.629951 1718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:00:03.630705 1733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.630860 1718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.630949 1718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "307963cba9764143a83cd3d5d409eb19"
format_stamp: "Formatted at 2026-09-25 19:00:03 on dist-test-slave-49bz"
server_key: "ca02b08aa5ee1ff700a4ed3911c0d9cb"
server_key_iv: "f04af0c358d3a4f6c039c2e59dc8cf8a"
server_key_version: "encryptionkey@0"
I20260925 19:00:03.631021 1718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:03.647966 1718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:03.649619 1718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:03.649765 1718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:03.650207 1718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:03.650285 1718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.650338 1718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:03.650367 1718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:00:03 dist-test-slave-49bz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:03.656978 1718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35395
I20260925 19:00:03.657045 1848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35395 every 8 connection(s)
I20260925 19:00:03.657287 1718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:00:03.659961 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.650646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55559 (local address 127.0.98.62:37745)
0925 19:00:03.650707 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.650709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.650945 (+ 236us) server_negotiation.cc:407] Connection header received
0925 19:00:03.651591 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.651592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.651601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.651686 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:03.652523 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.653314 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.654873 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.655079 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.655166 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.657163 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:03.657164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:03.657165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:03.657171 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:03.658599 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.659016 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.659018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.659018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.659082 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.659322 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.659323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.659323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.659435 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:03.659508 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.659778 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.659875 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:00:03.660117 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.650626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37745 (local address 127.0.98.2:55559)
0925 19:00:03.650777 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.650841 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.651492 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.651756 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.651761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.652133 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:03.652412 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.652418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.653405 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.653406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.654786 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.654789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.655177 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.655178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.655306 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.655791 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:03.655799 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:03.657088 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:03.658700 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.658706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.658708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.658927 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.659151 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.659152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.659152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.659248 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.659500 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:03.659506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:03.659636 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.659757 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.659911 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260925 19:00:03.660614 1849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37745
I20260925 19:00:03.660714 1849 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:03.660878 1849 heartbeater.cc:511] Master 127.0.98.62:37745 requested a full tablet report, sending...
I20260925 19:00:03.661247 1505 ts_manager.cc:194] Registered new tserver with Master: 307963cba9764143a83cd3d5d409eb19 (127.0.98.2:35395)
I20260925 19:00:03.661727 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55559
I20260925 19:00:03.663101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 1718
I20260925 19:00:03.663189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:00:03.663344 392 external_mini_cluster.cc:1428] Setting key e0289aa08fc435dd2a8ec7133beaf3e1
I20260925 19:00:03.667280 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.661494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37745 (local address 127.0.98.2:48009)
0925 19:00:03.661616 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.661625 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.661676 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.661783 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.661783 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.661964 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:03.662008 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.662009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.662705 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.662706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.663394 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.663396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.663768 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.663769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.663832 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.664265 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:03.664270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:03.664619 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:03.666371 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.666372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.666372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.666529 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.666730 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.666730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.666731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.666776 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.667086 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:03.667087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:03.667116 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.667160 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.667221 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260925 19:00:03.667367 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.661582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48009 (local address 127.0.98.62:37745)
0925 19:00:03.661619 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.661621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.661667 (+ 46us) server_negotiation.cc:407] Connection header received
0925 19:00:03.661725 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.661726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.661732 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.661769 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:03.662057 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.662632 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.663473 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.663709 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.663757 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.664711 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:03.664712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:03.664713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:03.664717 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:03.666293 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.666595 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.666596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.666596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.666651 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.666829 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.666830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.666830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.667015 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:03.667091 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.667184 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.667253 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:00:03.688634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37745
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:03.763701 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:03.763867 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:03.763907 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:03.764918 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:03.764950 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:03.764969 1858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:00:03.764985 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:03.766242 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:03.766429 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:03.766569 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:03.770927 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:03.770954 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:03.770954 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:03.771059 1858 server_base.cc:1034] running on GCE node
I20260925 19:00:03.771281 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:03.771514 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:03.772650 1858 hybrid_clock.cc:648] HybridClock initialized: now 1790362803772644 us; error 33 us; skew 500 ppm
Sep 25 19:00:03 dist-test-slave-49bz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:03.774963 1858 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:00:03.775317 1858 webserver.cc:460] Webserver started at http://127.0.98.3:39317/ using document root <none> and password file <none>
I20260925 19:00:03.775504 1858 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:03.775543 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:03.775656 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:03.776551 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "55d896df3ebf4499bef93656743bdcac"
format_stamp: "Formatted at 2026-09-25 19:00:03 on dist-test-slave-49bz"
server_key: "cb9e1eb14668b79679b008ecd51dfc5d"
server_key_iv: "dfec2fd0f2aa2a42785dfbecc8ee8131"
server_key_version: "encryptionkey@0"
I20260925 19:00:03.776865 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "55d896df3ebf4499bef93656743bdcac"
format_stamp: "Formatted at 2026-09-25 19:00:03 on dist-test-slave-49bz"
server_key: "cb9e1eb14668b79679b008ecd51dfc5d"
server_key_iv: "dfec2fd0f2aa2a42785dfbecc8ee8131"
server_key_version: "encryptionkey@0"
I20260925 19:00:03.778051 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:03.778779 1873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.778961 1858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.779064 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "55d896df3ebf4499bef93656743bdcac"
format_stamp: "Formatted at 2026-09-25 19:00:03 on dist-test-slave-49bz"
server_key: "cb9e1eb14668b79679b008ecd51dfc5d"
server_key_iv: "dfec2fd0f2aa2a42785dfbecc8ee8131"
server_key_version: "encryptionkey@0"
I20260925 19:00:03.779140 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:03.786082 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:03.787643 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:03.787808 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:03.788246 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:03.788311 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.788353 1858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:03.788378 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:00:03 dist-test-slave-49bz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:03.795157 1858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46321
I20260925 19:00:03.795463 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:00:03.795455 1988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46321 every 8 connection(s)
I20260925 19:00:03.797854 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.788704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57575 (local address 127.0.98.62:37745)
0925 19:00:03.788771 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.788774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.788926 (+ 152us) server_negotiation.cc:407] Connection header received
0925 19:00:03.789303 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.789304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.789318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.789393 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:03.790079 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.790895 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.792025 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.792381 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.792464 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.794882 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:03.794884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:03.794884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:03.794892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:03.796465 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.796916 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.796917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.796917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.796972 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.797231 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.797232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.797232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.797342 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:03.797415 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.797673 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.797767 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:00:03.797991 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.788672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37745 (local address 127.0.98.3:57575)
0925 19:00:03.788797 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.788832 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.789224 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.789448 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.789451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.789717 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:03.789983 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.789988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.791023 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.791024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.791925 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.791927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.792475 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.792476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.792590 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.793019 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:03.793031 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:03.794796 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:03.796559 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.796565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.796568 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.796830 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.797049 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.797050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.797050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.797166 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.797417 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:03.797423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:03.797544 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.797636 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.797789 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260925 19:00:03.798422 1989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37745
I20260925 19:00:03.798524 1989 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:03.798679 1989 heartbeater.cc:511] Master 127.0.98.62:37745 requested a full tablet report, sending...
I20260925 19:00:03.799067 1504 ts_manager.cc:194] Registered new tserver with Master: 55d896df3ebf4499bef93656743bdcac (127.0.98.3:46321)
I20260925 19:00:03.799496 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57575
I20260925 19:00:03.802415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 1858
I20260925 19:00:03.802532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:00:03.802692 392 external_mini_cluster.cc:1428] Setting key e1b4349b6c429dbc539a22c6ff37d677
I20260925 19:00:03.803439 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:00:03.804307 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.799313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37745 (local address 127.0.98.3:55567)
0925 19:00:03.799425 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.799434 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.799484 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.799729 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.799730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.799907 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:03.799957 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.799958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.800580 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.800580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.801033 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.801034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.801299 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.801299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.801338 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.801629 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:03.801633 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:03.801889 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:03.803535 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.803535 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.803536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.803684 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.803858 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:03.803859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:03.803859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:03.803900 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:03.804112 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:03.804113 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:03.804139 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.804195 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.804252 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260925 19:00:03.804399 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.799575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55567 (local address 127.0.98.62:37745)
0925 19:00:03.799626 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.799628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.799636 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:03.799654 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.799654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.799663 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.799711 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:03.800033 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.800501 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.801094 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.801243 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.801286 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.801965 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:03.801966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:03.801967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:03.801971 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:03.803465 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.803745 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.803746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.803746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.803799 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.803936 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.803936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.803936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.804045 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:03.804106 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.804217 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.804282 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 19:00:03 dist-test-slave-49bz krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:00:03 dist-test-slave-49bz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:03.824409 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.818002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.0.98.62:37745)
0925 19:00:03.818059 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.818061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.818070 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:03.818130 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.818131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.818140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.818211 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:03.818791 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.819261 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.820086 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.820264 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.820332 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.822185 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:03.822187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:03.822187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:03.822194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:03.823355 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.823680 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.823681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.823682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.823734 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:03.823914 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:03.823915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:03.823916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:03.824025 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:03.824145 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.824236 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.824311 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:00:03.825840 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:00:03.827502 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:03.834071 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.829752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34363 (local address 127.0.0.1:54004)
0925 19:00:03.829942 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.829980 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.830067 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.830455 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.830469 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.830471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:03.830536 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.830538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.831721 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.831722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.832848 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.832850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.833795 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.833796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.833855 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.833858 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.833936 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.833980 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":10,"thread_start_us":78,"threads_started":1}
I20260925 19:00:03.834074 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.830370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.0.98.3:46321)
0925 19:00:03.830596 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.830600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.830609 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:03.830644 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.830646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.830661 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.830734 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:03.830837 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.831475 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.832955 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.833522 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.833606 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.833691 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.833905 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.833959 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260925 19:00:03.834117 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.829836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54004 (local address 127.0.98.1:34363)
0925 19:00:03.830130 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.830135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.830146 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:00:03.830185 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.830187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.830204 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.830387 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:03.830611 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.831641 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.832934 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.833627 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.833692 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.833761 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.833954 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.834013 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":69,"threads_started":1}
I20260925 19:00:03.835083 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.829873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46321 (local address 127.0.0.1:34634)
0925 19:00:03.830093 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.830105 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.830157 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.830731 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.830732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.830734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:03.830785 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.830786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.831572 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.831573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.832848 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.832850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.833640 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.833642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.833723 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.833736 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.833823 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.835020 (+ 1197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":1147,"thread_start_us":89,"threads_started":1}
I20260925 19:00:03.834998 1643 tablet_service.cc:1467] Processing CreateTablet for tablet 10c6baefe6a64fe8b15e58a0762a0da3 (DEFAULT_TABLE table=test-table [id=f7d62ae2bf6a49c79476dba69eb68706]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:03.835317 1643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c6baefe6a64fe8b15e58a0762a0da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:03.835666 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 10c6baefe6a64fe8b15e58a0762a0da3 (DEFAULT_TABLE table=test-table [id=f7d62ae2bf6a49c79476dba69eb68706]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:03.835986 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c6baefe6a64fe8b15e58a0762a0da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:03.836002 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.829873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35395 (local address 127.0.0.1:37920)
0925 19:00:03.830230 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.830239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.832317 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.832577 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.832578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.832580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:03.832647 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.832649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.833777 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.833778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.834929 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.834931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.835767 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.835768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.835835 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.835838 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.835902 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.835934 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260925 19:00:03.836045 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.830127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.0.98.2:35395)
0925 19:00:03.830303 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.830308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.830321 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:00:03.832428 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.832431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.832446 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.832514 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:03.832721 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.833717 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.835015 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.835682 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.835753 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.835814 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.835915 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.835955 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260925 19:00:03.836638 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 10c6baefe6a64fe8b15e58a0762a0da3 (DEFAULT_TABLE table=test-table [id=f7d62ae2bf6a49c79476dba69eb68706]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:03.836944 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c6baefe6a64fe8b15e58a0762a0da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:03.838946 2012 tablet_bootstrap.cc:492] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: Bootstrap starting.
I20260925 19:00:03.839568 2012 tablet_bootstrap.cc:654] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:03.839556 2013 tablet_bootstrap.cc:492] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: Bootstrap starting.
I20260925 19:00:03.839920 2012 log.cc:824] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:03.840178 2013 tablet_bootstrap.cc:654] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:03.840528 2013 log.cc:824] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:03.840638 2015 tablet_bootstrap.cc:492] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: Bootstrap starting.
I20260925 19:00:03.840896 2012 tablet_bootstrap.cc:492] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: No bootstrap required, opened a new log
I20260925 19:00:03.841004 2012 ts_tablet_manager.cc:1397] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:00:03.841239 2015 tablet_bootstrap.cc:654] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:03.841398 2013 tablet_bootstrap.cc:492] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: No bootstrap required, opened a new log
I20260925 19:00:03.841460 2013 ts_tablet_manager.cc:1397] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:03.841562 2015 log.cc:824] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:03.842234 2015 tablet_bootstrap.cc:492] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: No bootstrap required, opened a new log
I20260925 19:00:03.842368 2015 ts_tablet_manager.cc:1397] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:03.842909 2012 raft_consensus.cc:348] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.843070 2012 raft_consensus.cc:374] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:03.843103 2012 raft_consensus.cc:729] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d896df3ebf4499bef93656743bdcac, State: Initialized, Role: FOLLOWER
I20260925 19:00:03.843233 2012 consensus_queue.cc:260] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.843321 2013 raft_consensus.cc:348] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.843469 2013 raft_consensus.cc:374] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:03.843490 2012 ts_tablet_manager.cc:1428] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:00:03.843564 2013 raft_consensus.cc:729] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5eaec249634478a14f05171b04feee, State: Initialized, Role: FOLLOWER
I20260925 19:00:03.843591 1989 heartbeater.cc:503] Master 127.0.98.62:37745 was elected leader, sending a full tablet report...
I20260925 19:00:03.843714 2013 consensus_queue.cc:260] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.843907 2015 raft_consensus.cc:348] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.844010 2013 ts_tablet_manager.cc:1428] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:00:03.844017 1709 heartbeater.cc:503] Master 127.0.98.62:37745 was elected leader, sending a full tablet report...
I20260925 19:00:03.844074 2015 raft_consensus.cc:374] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:03.844129 2015 raft_consensus.cc:729] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307963cba9764143a83cd3d5d409eb19, State: Initialized, Role: FOLLOWER
I20260925 19:00:03.844225 2015 consensus_queue.cc:260] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.844465 2015 ts_tablet_manager.cc:1428] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:00:03.844477 1849 heartbeater.cc:503] Master 127.0.98.62:37745 was elected leader, sending a full tablet report...
I20260925 19:00:03.905315 2019 raft_consensus.cc:482] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:03.905424 2019 raft_consensus.cc:504] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.905725 2019 leader_election.cc:288] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 307963cba9764143a83cd3d5d409eb19 (127.0.98.2:35395), 55d896df3ebf4499bef93656743bdcac (127.0.98.3:46321)
W20260925 19:00:03.907898 1850 tablet.cc:2369] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:00:03.908783 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.906020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54331 (local address 127.0.98.3:46321)
0925 19:00:03.906059 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.906060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.906153 (+ 93us) server_negotiation.cc:407] Connection header received
0925 19:00:03.906211 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.906212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.906223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.906305 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:03.906477 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.906924 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.907910 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.908424 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.908486 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.908494 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.908674 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.908715 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:00:03.908783 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.905962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60883 (local address 127.0.98.2:35395)
0925 19:00:03.906012 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.906014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.906025 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:00:03.906055 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.906056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.906066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.906147 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:03.906330 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.906783 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.907804 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.908355 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.908441 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.908452 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.908674 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.908715 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:00:03.908809 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.905932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46321 (local address 127.0.98.1:54331)
0925 19:00:03.906096 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.906103 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.906162 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.906314 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.906314 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.906326 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:03.906403 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.906404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.907004 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.907005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.907720 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.907722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.908499 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.908499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.908559 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.908561 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.908619 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.908663 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":4,"thread_start_us":105,"threads_started":1}
I20260925 19:00:03.908809 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.905900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35395 (local address 127.0.98.1:60883)
0925 19:00:03.905944 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:03.905957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:03.906002 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:03.906165 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:03.906166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:03.906168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:03.906239 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.906240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.906865 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.906866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.907662 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:03.907665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.908444 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.908445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:03.908551 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.908555 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:00:03.908619 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:00:03.908663 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":48}
I20260925 19:00:03.909200 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c6baefe6a64fe8b15e58a0762a0da3" candidate_uuid: "9f5eaec249634478a14f05171b04feee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307963cba9764143a83cd3d5d409eb19" is_pre_election: true
I20260925 19:00:03.909235 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c6baefe6a64fe8b15e58a0762a0da3" candidate_uuid: "9f5eaec249634478a14f05171b04feee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55d896df3ebf4499bef93656743bdcac" is_pre_election: true
I20260925 19:00:03.909317 1803 raft_consensus.cc:2454] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f5eaec249634478a14f05171b04feee in term 0.
I20260925 19:00:03.909376 1943 raft_consensus.cc:2454] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f5eaec249634478a14f05171b04feee in term 0.
I20260925 19:00:03.909554 1595 leader_election.cc:302] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307963cba9764143a83cd3d5d409eb19, 9f5eaec249634478a14f05171b04feee; no voters:
I20260925 19:00:03.909713 2019 raft_consensus.cc:2781] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:03.909749 2019 raft_consensus.cc:482] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:03.909763 2019 raft_consensus.cc:3037] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:03.910588 2019 raft_consensus.cc:504] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.910727 2019 leader_election.cc:288] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [CANDIDATE]: Term 1 election: Requested vote from peers 307963cba9764143a83cd3d5d409eb19 (127.0.98.2:35395), 55d896df3ebf4499bef93656743bdcac (127.0.98.3:46321)
I20260925 19:00:03.910853 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c6baefe6a64fe8b15e58a0762a0da3" candidate_uuid: "9f5eaec249634478a14f05171b04feee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55d896df3ebf4499bef93656743bdcac"
I20260925 19:00:03.910841 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c6baefe6a64fe8b15e58a0762a0da3" candidate_uuid: "9f5eaec249634478a14f05171b04feee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307963cba9764143a83cd3d5d409eb19"
I20260925 19:00:03.910912 1803 raft_consensus.cc:3037] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:03.910912 1943 raft_consensus.cc:3037] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:03.911675 1943 raft_consensus.cc:2454] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f5eaec249634478a14f05171b04feee in term 1.
I20260925 19:00:03.911675 1803 raft_consensus.cc:2454] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f5eaec249634478a14f05171b04feee in term 1.
I20260925 19:00:03.911880 1595 leader_election.cc:302] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307963cba9764143a83cd3d5d409eb19, 9f5eaec249634478a14f05171b04feee; no voters:
I20260925 19:00:03.912019 2019 raft_consensus.cc:2781] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:03.912192 2019 raft_consensus.cc:686] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 1 LEADER]: Becoming Leader. State: Replica: 9f5eaec249634478a14f05171b04feee, State: Running, Role: LEADER
I20260925 19:00:03.912283 2019 consensus_queue.cc:237] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.913239 1505 catalog_manager.cc:5495] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f5eaec249634478a14f05171b04feee (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9f5eaec249634478a14f05171b04feee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } health_report { overall_health: UNKNOWN } } }
I20260925 19:00:03.924388 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:03.922297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.0.98.1:34363)
0925 19:00:03.922333 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:03.922335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:03.922362 (+ 27us) server_negotiation.cc:407] Connection header received
0925 19:00:03.922462 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:03.922480 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:03.922495 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:03.922544 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:03.922631 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.923050 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.923761 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:03.923937 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:03.923989 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:03.924152 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:03.924235 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:03.924288 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:03.924327 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 19:00:03.924341 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:00:03.925880 1943 raft_consensus.cc:1261] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac [term 1 FOLLOWER]: Refusing update from remote peer 9f5eaec249634478a14f05171b04feee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:03.925879 1803 raft_consensus.cc:1261] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19 [term 1 FOLLOWER]: Refusing update from remote peer 9f5eaec249634478a14f05171b04feee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:03.926209 2019 consensus_queue.cc:1035] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [LEADER]: Connected to new peer: Peer: permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:03.926355 2022 consensus_queue.cc:1035] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [LEADER]: Connected to new peer: Peer: permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:03.928419 2025 mvcc.cc:205] Tried to move back new op lower bound from 7333326044878032896 to 7333326044825370624. Current Snapshot: MvccSnapshot[applied={T|T < 7333326044878032896}]
I20260925 19:00:03.928635 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333326044878032896 to 7333326044825370624. Current Snapshot: MvccSnapshot[applied={T|T < 7333326044878032896}]
I20260925 19:00:03.929234 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333326044878032896 to 7333326044825370624. Current Snapshot: MvccSnapshot[applied={T|T < 7333326044878032896}]
I20260925 19:00:03.931697 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:03.931787 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:03.932487 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36: Sending DeleteTablet for 3 replicas of tablet 10c6baefe6a64fe8b15e58a0762a0da3
I20260925 19:00:03.932780 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c6baefe6a64fe8b15e58a0762a0da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34634
I20260925 19:00:03.932919 1643 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c6baefe6a64fe8b15e58a0762a0da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54004
I20260925 19:00:03.932994 2035 tablet_replica.cc:330] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: stopping tablet replica
I20260925 19:00:03.933005 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c6baefe6a64fe8b15e58a0762a0da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37920
I20260925 19:00:03.933081 2035 raft_consensus.cc:2229] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:03.933205 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:00:03.933346 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:03.933396 2037 tablet_replica.cc:330] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: stopping tablet replica
I20260925 19:00:03.933408 2035 raft_consensus.cc:2258] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:03.933471 2037 raft_consensus.cc:2229] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:03.933588 2037 raft_consensus.cc:2258] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:03.933871 2035 ts_tablet_manager.cc:1905] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:03.934095 2037 ts_tablet_manager.cc:1905] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:03.934784 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 70020d6bc0de40e1b307d8d988d43449 (DEFAULT_TABLE table=test-table [id=1a234aec1d914bbbae7621f664bb2882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:03.934875 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70020d6bc0de40e1b307d8d988d43449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:03.935612 2035 ts_tablet_manager.cc:1918] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:03.935688 2035 log.cc:1196] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/10c6baefe6a64fe8b15e58a0762a0da3
I20260925 19:00:03.935736 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 70020d6bc0de40e1b307d8d988d43449 (DEFAULT_TABLE table=test-table [id=1a234aec1d914bbbae7621f664bb2882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:03.935832 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70020d6bc0de40e1b307d8d988d43449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:03.935945 2035 ts_tablet_manager.cc:1939] T 10c6baefe6a64fe8b15e58a0762a0da3 P 55d896df3ebf4499bef93656743bdcac: Deleting consensus metadata
I20260925 19:00:03.936275 2036 tablet_replica.cc:330] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: stopping tablet replica
I20260925 19:00:03.936396 2036 raft_consensus.cc:2229] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:03.936553 2036 raft_consensus.cc:2258] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:03.936944 2036 ts_tablet_manager.cc:1905] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:03.936998 2012 tablet_bootstrap.cc:492] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: Bootstrap starting.
I20260925 19:00:03.937064 1490 catalog_manager.cc:4841] TS 55d896df3ebf4499bef93656743bdcac (127.0.98.3:46321): tablet 10c6baefe6a64fe8b15e58a0762a0da3 (table test-table [id=f7d62ae2bf6a49c79476dba69eb68706]) successfully deleted
I20260925 19:00:03.937217 1643 tablet_service.cc:1467] Processing CreateTablet for tablet 70020d6bc0de40e1b307d8d988d43449 (DEFAULT_TABLE table=test-table [id=1a234aec1d914bbbae7621f664bb2882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:03.937317 1643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70020d6bc0de40e1b307d8d988d43449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:03.937368 2012 tablet_bootstrap.cc:654] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:03.937886 2012 tablet_bootstrap.cc:492] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: No bootstrap required, opened a new log
I20260925 19:00:03.937943 2012 ts_tablet_manager.cc:1397] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:03.938079 2012 raft_consensus.cc:348] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.938148 2012 raft_consensus.cc:374] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:03.938174 2012 raft_consensus.cc:729] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d896df3ebf4499bef93656743bdcac, State: Initialized, Role: FOLLOWER
I20260925 19:00:03.938213 2012 consensus_queue.cc:260] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.938318 2012 ts_tablet_manager.cc:1428] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:03.938963 2013 tablet_bootstrap.cc:492] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: Bootstrap starting.
I20260925 19:00:03.939150 2037 ts_tablet_manager.cc:1918] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:03.939211 2037 log.cc:1196] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/10c6baefe6a64fe8b15e58a0762a0da3
I20260925 19:00:03.939399 2013 tablet_bootstrap.cc:654] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:03.939471 2037 ts_tablet_manager.cc:1939] T 10c6baefe6a64fe8b15e58a0762a0da3 P 307963cba9764143a83cd3d5d409eb19: Deleting consensus metadata
I20260925 19:00:03.939927 1491 catalog_manager.cc:4841] TS 307963cba9764143a83cd3d5d409eb19 (127.0.98.2:35395): tablet 10c6baefe6a64fe8b15e58a0762a0da3 (table test-table [id=f7d62ae2bf6a49c79476dba69eb68706]) successfully deleted
I20260925 19:00:03.939962 2013 tablet_bootstrap.cc:492] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: No bootstrap required, opened a new log
I20260925 19:00:03.939965 2015 tablet_bootstrap.cc:492] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19: Bootstrap starting.
I20260925 19:00:03.940002 2013 ts_tablet_manager.cc:1397] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:03.940117 2013 raft_consensus.cc:348] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.940171 2013 raft_consensus.cc:374] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:03.940191 2013 raft_consensus.cc:729] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5eaec249634478a14f05171b04feee, State: Initialized, Role: FOLLOWER
I20260925 19:00:03.940239 2013 consensus_queue.cc:260] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.940315 2013 ts_tablet_manager.cc:1428] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.940330 2015 tablet_bootstrap.cc:654] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:03.941243 2015 tablet_bootstrap.cc:492] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19: No bootstrap required, opened a new log
I20260925 19:00:03.941299 2015 ts_tablet_manager.cc:1397] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:03.941424 2015 raft_consensus.cc:348] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.941488 2015 raft_consensus.cc:374] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:03.941516 2015 raft_consensus.cc:729] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307963cba9764143a83cd3d5d409eb19, State: Initialized, Role: FOLLOWER
I20260925 19:00:03.941560 2015 consensus_queue.cc:260] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:03.941629 2015 ts_tablet_manager.cc:1428] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:03.942168 2036 ts_tablet_manager.cc:1918] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:03.942245 2036 log.cc:1196] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/10c6baefe6a64fe8b15e58a0762a0da3
I20260925 19:00:03.942466 2036 ts_tablet_manager.cc:1939] T 10c6baefe6a64fe8b15e58a0762a0da3 P 9f5eaec249634478a14f05171b04feee: Deleting consensus metadata
I20260925 19:00:03.942806 1492 catalog_manager.cc:4841] TS 9f5eaec249634478a14f05171b04feee (127.0.98.1:34363): tablet 10c6baefe6a64fe8b15e58a0762a0da3 (table test-table [id=f7d62ae2bf6a49c79476dba69eb68706]) successfully deleted
W20260925 19:00:04.013338 1710 tablet.cc:2369] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:00:04.046195 1990 tablet.cc:2369] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:00:04.107230 2020 raft_consensus.cc:482] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:04.107353 2020 raft_consensus.cc:504] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.107743 2020 leader_election.cc:288] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f5eaec249634478a14f05171b04feee (127.0.98.1:34363), 55d896df3ebf4499bef93656743bdcac (127.0.98.3:46321)
I20260925 19:00:04.110378 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.107954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34363 (local address 127.0.98.2:47137)
0925 19:00:04.107996 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:04.108009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:04.108059 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:04.108205 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:04.108206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:04.108208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:04.108281 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.108282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.108925 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.108926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.109612 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.109614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.110097 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.110098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.110162 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.110165 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:04.110229 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:00:04.110298 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 19:00:04.110387 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.108029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47137 (local address 127.0.98.1:34363)
0925 19:00:04.108075 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.108077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.108086 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:04.108118 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.108119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.108128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.108202 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:04.108352 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.108828 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.109696 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.110025 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.110099 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.110108 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.110245 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.110321 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:00:04.110589 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.107983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46321 (local address 127.0.98.2:47847)
0925 19:00:04.108114 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:04.108124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:04.108206 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:04.108349 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:04.108350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:04.108351 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:04.108447 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.108448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.109110 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.109111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.109781 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.109783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.110314 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.110315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.110392 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.110395 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:04.110457 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 19:00:04.110499 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260925 19:00:04.110674 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.108052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47847 (local address 127.0.98.3:46321)
0925 19:00:04.108094 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.108096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.108195 (+ 99us) server_negotiation.cc:407] Connection header received
0925 19:00:04.108270 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.108271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.108285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.108346 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:04.108518 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.109031 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.109877 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.110182 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.110258 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.110298 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.110570 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.110609 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:00:04.110652 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70020d6bc0de40e1b307d8d988d43449" candidate_uuid: "307963cba9764143a83cd3d5d409eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f5eaec249634478a14f05171b04feee" is_pre_election: true
I20260925 19:00:04.110793 1663 raft_consensus.cc:2454] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 307963cba9764143a83cd3d5d409eb19 in term 0.
I20260925 19:00:04.110845 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70020d6bc0de40e1b307d8d988d43449" candidate_uuid: "307963cba9764143a83cd3d5d409eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55d896df3ebf4499bef93656743bdcac" is_pre_election: true
I20260925 19:00:04.110920 1943 raft_consensus.cc:2454] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 307963cba9764143a83cd3d5d409eb19 in term 0.
I20260925 19:00:04.110955 1737 leader_election.cc:302] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307963cba9764143a83cd3d5d409eb19, 9f5eaec249634478a14f05171b04feee; no voters:
I20260925 19:00:04.111097 2020 raft_consensus.cc:2781] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:04.111140 2020 raft_consensus.cc:482] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:04.111166 2020 raft_consensus.cc:3037] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:04.111788 2020 raft_consensus.cc:504] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.111925 2020 leader_election.cc:288] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [CANDIDATE]: Term 1 election: Requested vote from peers 9f5eaec249634478a14f05171b04feee (127.0.98.1:34363), 55d896df3ebf4499bef93656743bdcac (127.0.98.3:46321)
I20260925 19:00:04.112033 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70020d6bc0de40e1b307d8d988d43449" candidate_uuid: "307963cba9764143a83cd3d5d409eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f5eaec249634478a14f05171b04feee"
I20260925 19:00:04.112097 1663 raft_consensus.cc:3037] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:04.112082 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70020d6bc0de40e1b307d8d988d43449" candidate_uuid: "307963cba9764143a83cd3d5d409eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55d896df3ebf4499bef93656743bdcac"
I20260925 19:00:04.112176 1943 raft_consensus.cc:3037] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:04.112705 1663 raft_consensus.cc:2454] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307963cba9764143a83cd3d5d409eb19 in term 1.
I20260925 19:00:04.112808 1943 raft_consensus.cc:2454] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307963cba9764143a83cd3d5d409eb19 in term 1.
I20260925 19:00:04.112887 1737 leader_election.cc:302] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307963cba9764143a83cd3d5d409eb19, 9f5eaec249634478a14f05171b04feee; no voters:
I20260925 19:00:04.112983 2020 raft_consensus.cc:2781] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:04.113108 2020 raft_consensus.cc:686] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 1 LEADER]: Becoming Leader. State: Replica: 307963cba9764143a83cd3d5d409eb19, State: Running, Role: LEADER
I20260925 19:00:04.113173 2020 consensus_queue.cc:237] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.113795 1505 catalog_manager.cc:5495] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 307963cba9764143a83cd3d5d409eb19 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "307963cba9764143a83cd3d5d409eb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } health_report { overall_health: UNKNOWN } } }
Sep 25 19:00:04 dist-test-slave-49bz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:04.154138 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.142811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37745 (local address 127.0.0.1:47164)
0925 19:00:04.142978 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:04.142993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:04.143066 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:04.143225 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:04.143226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:04.143525 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:04.143563 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.143565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.144253 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.144254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.144746 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.144748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.145108 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.145108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.145169 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.150854 (+ 5685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:04.150858 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:04.152082 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:04.153339 (+ 1257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:04.153344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:04.153346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:04.153513 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:04.153665 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:04.153666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:04.153666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:04.153714 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:04.153889 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:04.153899 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:04.153933 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 19:00:04.153978 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 19:00:04.154048 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260925 19:00:04.154172 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.142872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.0.98.62:37745)
0925 19:00:04.142956 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.142958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.143049 (+ 91us) server_negotiation.cc:407] Connection header received
0925 19:00:04.143132 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.143132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.143142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.143219 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:04.143652 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.144165 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.144823 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.145010 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.145107 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.152156 (+ 7049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:04.152157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:04.152157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:04.152163 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:04.153275 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:04.153570 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:04.153570 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:04.153571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:04.153615 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:04.153760 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:04.153760 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:04.153760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:04.153843 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:04.153886 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.154005 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.154088 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":56}
I20260925 19:00:04.155776 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47164:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 19:00:04.155946 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:04.157087 1643 tablet_service.cc:1467] Processing CreateTablet for tablet fdd9319a073d425c99753aace2f4da6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5d2f36a8e68141b38a93ee6c27db12c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:00:04.157136 1923 tablet_service.cc:1467] Processing CreateTablet for tablet fdd9319a073d425c99753aace2f4da6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5d2f36a8e68141b38a93ee6c27db12c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:00:04.157128 1783 tablet_service.cc:1467] Processing CreateTablet for tablet fdd9319a073d425c99753aace2f4da6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5d2f36a8e68141b38a93ee6c27db12c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:00:04.157249 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd9319a073d425c99753aace2f4da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:04.157250 1643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd9319a073d425c99753aace2f4da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:04.157266 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd9319a073d425c99753aace2f4da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:04.158345 2015 tablet_bootstrap.cc:492] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: Bootstrap starting.
I20260925 19:00:04.158658 2012 tablet_bootstrap.cc:492] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: Bootstrap starting.
I20260925 19:00:04.158718 2015 tablet_bootstrap.cc:654] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:04.159487 2012 tablet_bootstrap.cc:654] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:04.159734 2015 tablet_bootstrap.cc:492] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: No bootstrap required, opened a new log
I20260925 19:00:04.159775 2015 ts_tablet_manager.cc:1397] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:04.159778 2013 tablet_bootstrap.cc:492] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: Bootstrap starting.
I20260925 19:00:04.159904 2015 raft_consensus.cc:348] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.159963 2015 raft_consensus.cc:374] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:04.159986 2015 raft_consensus.cc:729] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307963cba9764143a83cd3d5d409eb19, State: Initialized, Role: FOLLOWER
I20260925 19:00:04.160049 2015 consensus_queue.cc:260] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.160079 2012 tablet_bootstrap.cc:492] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: No bootstrap required, opened a new log
I20260925 19:00:04.160091 2013 tablet_bootstrap.cc:654] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:04.160116 2012 ts_tablet_manager.cc:1397] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:04.160144 2015 ts_tablet_manager.cc:1428] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:04.160188 2020 tablet_replica.cc:439] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } } }
I20260925 19:00:04.160265 2020 tablet_replica.cc:442] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:04.160249 2012 raft_consensus.cc:348] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.160301 2012 raft_consensus.cc:374] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:04.160336 2012 raft_consensus.cc:729] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d896df3ebf4499bef93656743bdcac, State: Initialized, Role: FOLLOWER
I20260925 19:00:04.160372 2012 consensus_queue.cc:260] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.160442 2012 ts_tablet_manager.cc:1428] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:04.160504 2018 tablet_replica.cc:439] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } } }
I20260925 19:00:04.160595 2018 tablet_replica.cc:442] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:04.161013 2013 tablet_bootstrap.cc:492] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: No bootstrap required, opened a new log
I20260925 19:00:04.161072 2013 ts_tablet_manager.cc:1397] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:04.161193 2013 raft_consensus.cc:348] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.161259 2013 raft_consensus.cc:374] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:04.161289 2013 raft_consensus.cc:729] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5eaec249634478a14f05171b04feee, State: Initialized, Role: FOLLOWER
I20260925 19:00:04.161357 2013 consensus_queue.cc:260] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.161454 2013 ts_tablet_manager.cc:1428] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:04.161499 2030 tablet_replica.cc:439] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } } }
I20260925 19:00:04.161602 2030 tablet_replica.cc:442] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:04.235998 2018 raft_consensus.cc:482] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:04.236120 2018 raft_consensus.cc:504] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.236524 2018 leader_election.cc:288] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f5eaec249634478a14f05171b04feee (127.0.98.1:34363), 307963cba9764143a83cd3d5d409eb19 (127.0.98.2:35395)
I20260925 19:00:04.239532 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.236731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41973 (local address 127.0.98.1:34363)
0925 19:00:04.236780 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.236782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.236816 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:00:04.236845 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.236846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.236857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.236919 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:04.237105 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.237647 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.238693 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.239218 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.239267 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.239275 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.239457 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.239482 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:00:04.239557 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.236671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34363 (local address 127.0.98.3:41973)
0925 19:00:04.236723 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:04.236735 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:04.236777 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:04.236949 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:04.236949 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:04.236952 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:04.237031 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.237033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.237731 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.237732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.238611 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.238614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.239290 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.239291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.239369 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.239372 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:04.239448 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:00:04.239485 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 19:00:04.239806 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd9319a073d425c99753aace2f4da6e" candidate_uuid: "55d896df3ebf4499bef93656743bdcac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f5eaec249634478a14f05171b04feee" is_pre_election: true
I20260925 19:00:04.239913 1663 raft_consensus.cc:2454] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d896df3ebf4499bef93656743bdcac in term 0.
I20260925 19:00:04.239923 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.236773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58009 (local address 127.0.98.2:35395)
0925 19:00:04.236813 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.236815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.236934 (+ 119us) server_negotiation.cc:407] Connection header received
0925 19:00:04.236982 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.236983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.236991 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.237050 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:04.237225 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.237725 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.239100 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.239594 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.239658 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.239667 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.239831 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.239868 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:04.239929 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.236686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35395 (local address 127.0.98.3:58009)
0925 19:00:04.236867 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:04.236877 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:04.236929 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:04.237079 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:04.237080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:04.237082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:04.237158 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.237160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.237811 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.237812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.239012 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.239015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.239697 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.239698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.239766 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.239770 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:00:04.239815 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 19:00:04.239849 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":102,"threads_started":1}
I20260925 19:00:04.240136 1877 leader_election.cc:302] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55d896df3ebf4499bef93656743bdcac, 9f5eaec249634478a14f05171b04feee; no voters:
I20260925 19:00:04.240154 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd9319a073d425c99753aace2f4da6e" candidate_uuid: "55d896df3ebf4499bef93656743bdcac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307963cba9764143a83cd3d5d409eb19" is_pre_election: true
I20260925 19:00:04.240228 1803 raft_consensus.cc:2454] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d896df3ebf4499bef93656743bdcac in term 0.
I20260925 19:00:04.240281 2018 raft_consensus.cc:2781] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:04.240334 2018 raft_consensus.cc:482] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:04.240360 2018 raft_consensus.cc:3037] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:04.240998 2018 raft_consensus.cc:504] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.241132 2018 leader_election.cc:288] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [CANDIDATE]: Term 1 election: Requested vote from peers 9f5eaec249634478a14f05171b04feee (127.0.98.1:34363), 307963cba9764143a83cd3d5d409eb19 (127.0.98.2:35395)
I20260925 19:00:04.241263 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd9319a073d425c99753aace2f4da6e" candidate_uuid: "55d896df3ebf4499bef93656743bdcac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f5eaec249634478a14f05171b04feee"
I20260925 19:00:04.241282 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd9319a073d425c99753aace2f4da6e" candidate_uuid: "55d896df3ebf4499bef93656743bdcac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307963cba9764143a83cd3d5d409eb19"
I20260925 19:00:04.241333 1663 raft_consensus.cc:3037] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:04.241335 1803 raft_consensus.cc:3037] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:04.241984 1663 raft_consensus.cc:2454] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d896df3ebf4499bef93656743bdcac in term 1.
I20260925 19:00:04.241984 1803 raft_consensus.cc:2454] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d896df3ebf4499bef93656743bdcac in term 1.
I20260925 19:00:04.242174 1876 leader_election.cc:302] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307963cba9764143a83cd3d5d409eb19, 55d896df3ebf4499bef93656743bdcac; no voters:
I20260925 19:00:04.242316 2018 raft_consensus.cc:2781] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:04.242455 2018 raft_consensus.cc:686] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [term 1 LEADER]: Becoming Leader. State: Replica: 55d896df3ebf4499bef93656743bdcac, State: Running, Role: LEADER
I20260925 19:00:04.242527 2018 consensus_queue.cc:237] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } }
I20260925 19:00:04.243208 2054 tablet_replica.cc:439] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: TxnStatusTablet state changed. Reason: New leader 55d896df3ebf4499bef93656743bdcac. Latest consensus state: current_term: 1 leader_uuid: "55d896df3ebf4499bef93656743bdcac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } } }
I20260925 19:00:04.243289 2054 tablet_replica.cc:442] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:00:04.243451 2056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 19:00:04.243577 2056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 19:00:04.243599 1505 catalog_manager.cc:5495] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac reported cstate change: term changed from 0 to 1, leader changed from <none> to 55d896df3ebf4499bef93656743bdcac (127.0.98.3). New cstate: current_term: 1 leader_uuid: "55d896df3ebf4499bef93656743bdcac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } health_report { overall_health: HEALTHY } } }
I20260925 19:00:04.254456 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.252249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.0.98.3:46321)
0925 19:00:04.252292 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.252293 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.252357 (+ 64us) server_negotiation.cc:407] Connection header received
0925 19:00:04.252441 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.252442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.252457 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.252526 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:04.252667 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.253074 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.253746 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.253915 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.253980 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.254147 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:04.254237 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:04.254328 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.254379 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.254393 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:00:04.254487 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.252169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46321 (local address 127.0.0.1:34640)
0925 19:00:04.252309 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:04.252319 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:04.252373 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:04.252531 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:04.252532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:04.252533 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:04.252590 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.252592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.253147 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.253148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.253665 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.253667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.253981 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.253982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.254042 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.254056 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:04.254315 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:04.254324 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 19:00:04.254374 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:00:04.254417 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 19:00:04.255414 1803 raft_consensus.cc:1261] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19 [term 1 FOLLOWER]: Refusing update from remote peer 55d896df3ebf4499bef93656743bdcac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:04.255494 1663 raft_consensus.cc:1261] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee [term 1 FOLLOWER]: Refusing update from remote peer 55d896df3ebf4499bef93656743bdcac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:04.255733 2054 consensus_queue.cc:1035] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:04.255837 2018 consensus_queue.cc:1035] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac [LEADER]: Connected to new peer: Peer: permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:04.256275 2020 tablet_replica.cc:439] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: TxnStatusTablet state changed. Reason: New leader 55d896df3ebf4499bef93656743bdcac. Latest consensus state: current_term: 1 leader_uuid: "55d896df3ebf4499bef93656743bdcac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } } }
I20260925 19:00:04.256357 2020 tablet_replica.cc:442] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:04.256846 2030 tablet_replica.cc:439] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: TxnStatusTablet state changed. Reason: New leader 55d896df3ebf4499bef93656743bdcac. Latest consensus state: current_term: 1 leader_uuid: "55d896df3ebf4499bef93656743bdcac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } } }
I20260925 19:00:04.256960 2030 tablet_replica.cc:442] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:04.257241 2018 tablet_replica.cc:439] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55d896df3ebf4499bef93656743bdcac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } } }
I20260925 19:00:04.257318 2018 tablet_replica.cc:442] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:00:04.257489 2018 tablet_replica.cc:439] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55d896df3ebf4499bef93656743bdcac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } } }
I20260925 19:00:04.257548 2018 tablet_replica.cc:442] T fdd9319a073d425c99753aace2f4da6e P 55d896df3ebf4499bef93656743bdcac: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:00:04.257915 2020 tablet_replica.cc:439] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55d896df3ebf4499bef93656743bdcac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } } }
I20260925 19:00:04.257984 2020 tablet_replica.cc:442] T fdd9319a073d425c99753aace2f4da6e P 307963cba9764143a83cd3d5d409eb19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:04.258004 2030 tablet_replica.cc:439] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55d896df3ebf4499bef93656743bdcac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 } } peers { permanent_uuid: "307963cba9764143a83cd3d5d409eb19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35395 } } peers { permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 } } }
I20260925 19:00:04.258074 2030 tablet_replica.cc:442] T fdd9319a073d425c99753aace2f4da6e P 9f5eaec249634478a14f05171b04feee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:04.261833 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.259373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.0.98.2:35395)
0925 19:00:04.259433 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.259435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.259442 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:00:04.259468 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.259468 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.259482 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.259532 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:04.259679 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.260106 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.261114 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.261276 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.261325 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.261479 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:04.261555 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:04.261602 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.261700 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.261718 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:00:04.266036 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.263926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32911 (local address 127.0.98.3:46321)
0925 19:00:04.263979 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.263981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.264103 (+ 122us) server_negotiation.cc:407] Connection header received
0925 19:00:04.264158 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.264158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.264166 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.264210 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:04.264299 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.264710 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.265365 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.265548 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.265609 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.265791 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:04.265818 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:04.265882 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.265946 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.265960 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:00:04.266086 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.263870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46321 (local address 127.0.98.2:32911)
0925 19:00:04.264045 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:04.264053 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:04.264116 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:04.264211 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:04.264212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:04.264212 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:04.264257 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.264259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.264783 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.264784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.265284 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.265285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.265620 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.265621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.265700 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.265710 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:04.265884 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:04.265887 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:04.265942 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 19:00:04.265994 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260925 19:00:04.268285 1663 raft_consensus.cc:1261] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee [term 1 FOLLOWER]: Refusing update from remote peer 307963cba9764143a83cd3d5d409eb19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:04.268468 1943 raft_consensus.cc:1261] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac [term 1 FOLLOWER]: Refusing update from remote peer 307963cba9764143a83cd3d5d409eb19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:04.268726 2042 consensus_queue.cc:1035] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f5eaec249634478a14f05171b04feee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:04.268841 2020 consensus_queue.cc:1035] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55d896df3ebf4499bef93656743bdcac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:04.277057 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.274332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47993 (local address 127.0.98.2:35395)
0925 19:00:04.274369 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.274371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.274511 (+ 140us) server_negotiation.cc:407] Connection header received
0925 19:00:04.274530 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.274531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.274540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.274584 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:04.274731 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.275168 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.276372 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.276641 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.276729 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.276887 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:04.276910 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:04.276949 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.277000 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.277014 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:00:04.277105 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.274335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35395 (local address 127.0.98.3:47993)
0925 19:00:04.274399 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:04.274409 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:04.274459 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:04.274616 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:04.274617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:04.274618 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:04.274673 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.274674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.275242 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.275243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.276206 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:04.276208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.276730 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.276730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:04.276795 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.276809 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:04.276951 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:04.276952 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:00:04.276999 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:00:04.277034 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 19:00:04.283663 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:04.283748 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:04.284343 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1445a9468a694150bcec064903d45c36: Sending DeleteTablet for 3 replicas of tablet 70020d6bc0de40e1b307d8d988d43449
I20260925 19:00:04.284582 1643 tablet_service.cc:1514] Processing DeleteTablet for tablet 70020d6bc0de40e1b307d8d988d43449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54004
I20260925 19:00:04.284612 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 70020d6bc0de40e1b307d8d988d43449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34634
I20260925 19:00:04.284683 2035 tablet_replica.cc:330] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: stopping tablet replica
I20260925 19:00:04.284678 2036 tablet_replica.cc:330] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: stopping tablet replica
I20260925 19:00:04.284723 2035 raft_consensus.cc:2229] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:04.284734 2036 raft_consensus.cc:2229] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:04.284756 2035 raft_consensus.cc:2258] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:04.284765 2036 raft_consensus.cc:2258] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:04.285073 2035 ts_tablet_manager.cc:1905] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:04.285079 2036 ts_tablet_manager.cc:1905] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:04.286336 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 70020d6bc0de40e1b307d8d988d43449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37920
I20260925 19:00:04.286449 2037 tablet_replica.cc:330] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19: stopping tablet replica
I20260925 19:00:04.286496 2037 raft_consensus.cc:2229] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:04.286583 2037 raft_consensus.cc:2258] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:04.286803 2036 ts_tablet_manager.cc:1918] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:04.286813 2035 ts_tablet_manager.cc:1918] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:04.286860 2035 log.cc:1196] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/70020d6bc0de40e1b307d8d988d43449
I20260925 19:00:04.286859 2036 log.cc:1196] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/70020d6bc0de40e1b307d8d988d43449
I20260925 19:00:04.287084 2035 ts_tablet_manager.cc:1939] T 70020d6bc0de40e1b307d8d988d43449 P 55d896df3ebf4499bef93656743bdcac: Deleting consensus metadata
I20260925 19:00:04.287092 2036 ts_tablet_manager.cc:1939] T 70020d6bc0de40e1b307d8d988d43449 P 9f5eaec249634478a14f05171b04feee: Deleting consensus metadata
I20260925 19:00:04.287379 1490 catalog_manager.cc:4841] TS 55d896df3ebf4499bef93656743bdcac (127.0.98.3:46321): tablet 70020d6bc0de40e1b307d8d988d43449 (table test-table [id=1a234aec1d914bbbae7621f664bb2882]) successfully deleted
I20260925 19:00:04.287405 2037 ts_tablet_manager.cc:1905] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:04.287402 1492 catalog_manager.cc:4841] TS 9f5eaec249634478a14f05171b04feee (127.0.98.1:34363): tablet 70020d6bc0de40e1b307d8d988d43449 (table test-table [id=1a234aec1d914bbbae7621f664bb2882]) successfully deleted
I20260925 19:00:04.288573 2037 ts_tablet_manager.cc:1918] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:04.288641 2037 log.cc:1196] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/70020d6bc0de40e1b307d8d988d43449
I20260925 19:00:04.288836 2037 ts_tablet_manager.cc:1939] T 70020d6bc0de40e1b307d8d988d43449 P 307963cba9764143a83cd3d5d409eb19: Deleting consensus metadata
I20260925 19:00:04.289151 1491 catalog_manager.cc:4841] TS 307963cba9764143a83cd3d5d409eb19 (127.0.98.2:35395): tablet 70020d6bc0de40e1b307d8d988d43449 (table test-table [id=1a234aec1d914bbbae7621f664bb2882]) successfully deleted
Sep 25 19:00:04 dist-test-slave-49bz krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 19:00:04.295905 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.285366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.0.98.1:34363)
0925 19:00:04.285407 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.285409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.285562 (+ 153us) server_negotiation.cc:407] Connection header received
0925 19:00:04.285608 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.285609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.285619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.285668 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:04.286398 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.287161 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.289897 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.290101 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.290187 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.292991 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:04.292998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:04.292998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:04.293005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:04.294613 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:04.294969 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:04.294972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:04.294973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:04.295031 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:04.295250 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:04.295251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:04.295252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:04.295379 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:04.295539 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.295713 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.295789 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 19:00:04.296114 1703 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54020
I20260925 19:00:04.303144 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:04.296926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.0.98.62:37745)
0925 19:00:04.296963 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:04.296965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:04.297134 (+ 169us) server_negotiation.cc:407] Connection header received
0925 19:00:04.297209 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:04.297209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:04.297220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:04.297273 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:04.297737 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.298411 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.299182 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:04.299403 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:04.299481 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:04.300622 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:04.300623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:04.300623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:04.300629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:04.301982 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:04.302362 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:04.302363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:04.302364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:04.302415 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:04.302625 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:04.302626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:04.302627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:04.302750 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:04.302847 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:04.302955 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 19:00:04.303030 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 19:00:04.303314 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47174
I20260925 19:00:04.303872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 1577
I20260925 19:00:04.310665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 1718
I20260925 19:00:04.315706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 1858
I20260925 19:00:04.320019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 1473
2026-09-25T19:00:04Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6275 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:04 dist-test-slave-49bz krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:04 dist-test-slave-49bz krb5kdc[2077](info): set up 2 sockets
Sep 25 19:00:04 dist-test-slave-49bz krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260925 19:00:05.325760 392 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:05 dist-test-slave-49bz krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:00:05.431200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:05.506276 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:05.506505 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:05.506538 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:05.507616 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:05.507682 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 19:00:05.507722 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:05.507745 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:05.507772 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:05.507797 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:05.509232 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36317
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:05.509481 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:05.509712 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:05.514298 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:05.514389 2090 server_base.cc:1034] running on GCE node
W20260925 19:00:05.514454 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:05.514356 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:05.514724 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:05.515038 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:05.516331 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790362805516315 us; error 25 us; skew 500 ppm
Sep 25 19:00:05 dist-test-slave-49bz krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:05.518925 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:00:05.519289 2090 webserver.cc:460] Webserver started at http://127.0.98.62:42727/ using document root <none> and password file <none>
I20260925 19:00:05.519476 2090 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:05.519515 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:05.519599 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:05.520552 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "7efcb76217354658b2a7044a0e0b9a1b"
format_stamp: "Formatted at 2026-09-25 19:00:05 on dist-test-slave-49bz"
server_key: "3a0e0a23939440d7854d187e1ba922a4"
server_key_iv: "391d0a7fff4865282f0bf0ffd24e2744"
server_key_version: "encryptionkey@0"
I20260925 19:00:05.520831 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "7efcb76217354658b2a7044a0e0b9a1b"
format_stamp: "Formatted at 2026-09-25 19:00:05 on dist-test-slave-49bz"
server_key: "3a0e0a23939440d7854d187e1ba922a4"
server_key_iv: "391d0a7fff4865282f0bf0ffd24e2744"
server_key_version: "encryptionkey@0"
I20260925 19:00:05.522043 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:00:05.522733 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:05.522876 2090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:05.522966 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "7efcb76217354658b2a7044a0e0b9a1b"
format_stamp: "Formatted at 2026-09-25 19:00:05 on dist-test-slave-49bz"
server_key: "3a0e0a23939440d7854d187e1ba922a4"
server_key_iv: "391d0a7fff4865282f0bf0ffd24e2744"
server_key_version: "encryptionkey@0"
I20260925 19:00:05.523020 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:05.546612 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:05.548352 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:05.548453 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:05.552467 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36317
I20260925 19:00:05.552542 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36317 every 8 connection(s)
I20260925 19:00:05.552821 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:05.553550 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:05.554904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 2090
I20260925 19:00:05.555009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:05.555176 392 external_mini_cluster.cc:1428] Setting key 10242009b9be6afdaf6732543183088e
I20260925 19:00:05.556041 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b: Bootstrap starting.
I20260925 19:00:05.556648 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:05.556965 2168 log.cc:824] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:05.557591 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b: No bootstrap required, opened a new log
I20260925 19:00:05.559803 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efcb76217354658b2a7044a0e0b9a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260925 19:00:05.559957 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:05.559981 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7efcb76217354658b2a7044a0e0b9a1b, State: Initialized, Role: FOLLOWER
I20260925 19:00:05.560184 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efcb76217354658b2a7044a0e0b9a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260925 19:00:05.560276 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:05.560307 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:05.560350 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:05.561012 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efcb76217354658b2a7044a0e0b9a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260925 19:00:05.561120 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7efcb76217354658b2a7044a0e0b9a1b; no voters:
I20260925 19:00:05.561286 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:05.561331 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:05.561492 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [term 1 LEADER]: Becoming Leader. State: Replica: 7efcb76217354658b2a7044a0e0b9a1b, State: Running, Role: LEADER
I20260925 19:00:05.561519 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:05.561599 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efcb76217354658b2a7044a0e0b9a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
Sep 25 19:00:05 dist-test-slave-49bz krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:05.561924 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7efcb76217354658b2a7044a0e0b9a1b. Latest consensus state: current_term: 1 leader_uuid: "7efcb76217354658b2a7044a0e0b9a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efcb76217354658b2a7044a0e0b9a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } } }
I20260925 19:00:05.562053 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:05.562242 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:05.562304 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7efcb76217354658b2a7044a0e0b9a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efcb76217354658b2a7044a0e0b9a1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } } }
I20260925 19:00:05.562412 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:05.562768 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:05.564273 2182 catalog_manager.cc:1347] Generated new cluster ID: b24d6faa99cc465286b536b9c294a5dd
I20260925 19:00:05.564320 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:05.565656 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:05.555657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.0.98.62:36317)
0925 19:00:05.555798 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:05.555809 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:05.555832 (+ 23us) server_negotiation.cc:407] Connection header received
0925 19:00:05.556235 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:05.556242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:05.556284 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:05.556386 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:05.557089 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.557952 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.559136 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.559535 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.559610 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:05.562303 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:05.562306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:05.562307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:05.562318 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:05.564222 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:05.564593 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:05.564596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:05.564596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:05.564681 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:05.564878 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:05.564879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:05.564879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:05.565119 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:05.565231 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:05.565363 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 19:00:05.565478 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260925 19:00:05.598981 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:05.599660 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:05.607839 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b: Generated new TSK 0
I20260925 19:00:05.608033 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:00:05.638211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36317
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:05.713439 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:05.713680 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:05.713712 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:05.714793 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:05.714855 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:05.714886 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 19:00:05.714920 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:00:05.714952 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:05.716236 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:05.716472 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:05.716671 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:00:05.721551 2194 server_base.cc:1034] running on GCE node
W20260925 19:00:05.721503 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:05.721509 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:05.721503 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:05.721971 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:05.722172 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:05.723313 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790362805723296 us; error 32 us; skew 500 ppm
Sep 25 19:00:05 dist-test-slave-49bz krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:05.725591 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:00:05.725956 2194 webserver.cc:460] Webserver started at http://127.0.98.1:39621/ using document root <none> and password file <none>
I20260925 19:00:05.726166 2194 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:05.726221 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:05.726373 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:05.727277 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "21f0cdf4ac7e43138e291e2e0a70bc93"
format_stamp: "Formatted at 2026-09-25 19:00:05 on dist-test-slave-49bz"
server_key: "86cbfce45db95361914b9fc8592d638e"
server_key_iv: "0c2bf41cfc36f607fbc21693d1e04c70"
server_key_version: "encryptionkey@0"
I20260925 19:00:05.727598 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "21f0cdf4ac7e43138e291e2e0a70bc93"
format_stamp: "Formatted at 2026-09-25 19:00:05 on dist-test-slave-49bz"
server_key: "86cbfce45db95361914b9fc8592d638e"
server_key_iv: "0c2bf41cfc36f607fbc21693d1e04c70"
server_key_version: "encryptionkey@0"
I20260925 19:00:05.728878 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:00:05.729635 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:05.729805 2194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:00:05.729897 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "21f0cdf4ac7e43138e291e2e0a70bc93"
format_stamp: "Formatted at 2026-09-25 19:00:05 on dist-test-slave-49bz"
server_key: "86cbfce45db95361914b9fc8592d638e"
server_key_iv: "0c2bf41cfc36f607fbc21693d1e04c70"
server_key_version: "encryptionkey@0"
I20260925 19:00:05.729974 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:05.744773 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:05.746517 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:05.746656 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:05.747074 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:05.747131 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:05.747170 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:05.747197 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:05.753594 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40503
I20260925 19:00:05.753651 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40503 every 8 connection(s)
Sep 25 19:00:05 dist-test-slave-49bz krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:05.753953 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:00:05.756906 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:05.747513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51823 (local address 127.0.98.62:36317)
0925 19:00:05.747580 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:05.747583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:05.747750 (+ 167us) server_negotiation.cc:407] Connection header received
0925 19:00:05.748245 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:05.748247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:05.748259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:05.748343 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:05.749163 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.749926 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.751168 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.751366 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.751453 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:05.754505 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:05.754506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:05.754507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:05.754513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:05.755640 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:05.755994 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:05.755996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:05.755997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:05.756053 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:05.756293 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:05.756294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:05.756294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:05.756400 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:05.756474 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:05.756738 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 19:00:05.756822 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:00:05.757055 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:05.747492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:51823)
0925 19:00:05.747635 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:05.747669 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:05.748151 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:05.748413 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:05.748417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:05.748758 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:05.749044 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:05.749051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.750035 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.750037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:05.751088 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:05.751090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.751472 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.751473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:05.751598 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:05.752707 (+ 1109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:05.752719 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:05.754406 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:05.755716 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:05.755722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:05.755724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:05.755924 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:05.756114 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:05.756115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:05.756115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:05.756217 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:05.756466 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:05.756472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:05.756612 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 19:00:05.756707 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 19:00:05.756844 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
I20260925 19:00:05.757506 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36317
I20260925 19:00:05.757578 2325 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:05.757730 2325 heartbeater.cc:511] Master 127.0.98.62:36317 requested a full tablet report, sending...
I20260925 19:00:05.758242 2121 ts_manager.cc:194] Registered new tserver with Master: 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1:40503)
I20260925 19:00:05.758953 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51823
I20260925 19:00:05.762284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 2194
I20260925 19:00:05.762392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:05.762593 392 external_mini_cluster.cc:1428] Setting key ace1d6ce7793794bbb61b5e2730749a4
I20260925 19:00:05.763401 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:05.758443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58535 (local address 127.0.98.62:36317)
0925 19:00:05.758475 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:05.758476 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:05.758601 (+ 125us) server_negotiation.cc:407] Connection header received
0925 19:00:05.758671 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:05.758672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:05.758681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:05.758734 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:05.759050 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.759517 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.760176 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.760387 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.760474 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:05.761218 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:05.761219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:05.761219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:05.761225 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:05.762398 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:05.762777 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:05.762778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:05.762778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:05.762838 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:05.762976 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:05.762977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:05.762977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:05.763086 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:05.763149 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:05.763265 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 19:00:05.763326 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 19:00:05.763427 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:05.758404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:58535)
0925 19:00:05.758527 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:05.758536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:05.758615 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:05.758739 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:05.758739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:05.758912 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:05.758967 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:05.758968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.759603 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.759603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:05.760101 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:05.760102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.760476 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.760476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:05.760550 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:05.760871 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:05.760874 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:05.761139 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:05.762499 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:05.762499 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:05.762500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:05.762701 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:05.762892 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:05.762893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:05.762893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:05.762939 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:05.763139 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:05.763140 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:05.763169 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 19:00:05.763237 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:00:05.763322 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:00:05.784957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36317
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:05.862361 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:05.862582 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:05.862608 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:05.863744 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:05.863785 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:05.863806 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 19:00:05.863852 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:00:05.863878 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:05.865326 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:05.865550 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:05.865769 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:05.870589 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:05.870623 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:05.870615 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:05.870680 2334 server_base.cc:1034] running on GCE node
I20260925 19:00:05.870978 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:05.871204 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:05.872344 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790362805872332 us; error 30 us; skew 500 ppm
Sep 25 19:00:05 dist-test-slave-49bz krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:05.874734 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:00:05.875082 2334 webserver.cc:460] Webserver started at http://127.0.98.2:43809/ using document root <none> and password file <none>
I20260925 19:00:05.875239 2334 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:05.875275 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:05.875355 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:05.876286 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "589e62e73d3e42d4ac54b4d884a29c69"
format_stamp: "Formatted at 2026-09-25 19:00:05 on dist-test-slave-49bz"
server_key: "4d9be554b642bcb1817bfb5c90a57238"
server_key_iv: "4b04622ca48075bfb9605bf05785caec"
server_key_version: "encryptionkey@0"
I20260925 19:00:05.876582 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "589e62e73d3e42d4ac54b4d884a29c69"
format_stamp: "Formatted at 2026-09-25 19:00:05 on dist-test-slave-49bz"
server_key: "4d9be554b642bcb1817bfb5c90a57238"
server_key_iv: "4b04622ca48075bfb9605bf05785caec"
server_key_version: "encryptionkey@0"
I20260925 19:00:05.877801 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:00:05.878600 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:05.878773 2334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:05.878852 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "589e62e73d3e42d4ac54b4d884a29c69"
format_stamp: "Formatted at 2026-09-25 19:00:05 on dist-test-slave-49bz"
server_key: "4d9be554b642bcb1817bfb5c90a57238"
server_key_iv: "4b04622ca48075bfb9605bf05785caec"
server_key_version: "encryptionkey@0"
I20260925 19:00:05.878932 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:05.891320 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:05.893339 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:05.893463 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:05.893858 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:05.893896 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:05.893947 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:05.893998 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:05.900408 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42591
I20260925 19:00:05.900484 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42591 every 8 connection(s)
I20260925 19:00:05.900765 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:00:05 dist-test-slave-49bz krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:05.903649 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:05.894388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44515 (local address 127.0.98.62:36317)
0925 19:00:05.894468 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:05.894470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:05.894762 (+ 292us) server_negotiation.cc:407] Connection header received
0925 19:00:05.895181 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:05.895183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:05.895195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:05.895271 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:05.896030 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.896850 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.898382 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.898707 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.898795 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:05.901367 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:05.901368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:05.901368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:05.901393 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:05.902649 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:05.902971 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:05.902972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:05.902972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:05.903016 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:05.903233 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:05.903233 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:05.903233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:05.903318 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:05.903360 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:05.903541 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 19:00:05.903593 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 19:00:05.903820 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:05.894382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:44515)
0925 19:00:05.894624 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:05.894660 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:05.895085 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:05.895346 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:05.895350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:05.895659 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:05.895915 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:05.895919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.896963 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.896964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:05.898292 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:05.898294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.898863 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.898865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:05.899005 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:05.899608 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:05.899618 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:05.901283 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:05.902710 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:05.902715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:05.902716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:05.902904 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:05.903085 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:05.903085 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:05.903086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:05.903171 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:05.903365 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:05.903368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:05.903462 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 19:00:05.903530 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:00:05.903630 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":176,"threads_started":1}
I20260925 19:00:05.904271 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36317
I20260925 19:00:05.904345 2465 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:05.904524 2465 heartbeater.cc:511] Master 127.0.98.62:36317 requested a full tablet report, sending...
I20260925 19:00:05.904904 2122 ts_manager.cc:194] Registered new tserver with Master: 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591)
I20260925 19:00:05.905339 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44515
I20260925 19:00:05.909091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 2334
I20260925 19:00:05.909183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:00:05.909363 392 external_mini_cluster.cc:1428] Setting key 67b1cf7e9c68969bab51d176ba8f5812
I20260925 19:00:05.910360 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:05.905266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45913 (local address 127.0.98.62:36317)
0925 19:00:05.905303 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:05.905304 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:05.905424 (+ 120us) server_negotiation.cc:407] Connection header received
0925 19:00:05.905476 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:05.905476 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:05.905486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:05.905550 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:05.905865 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.906302 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.906858 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.907055 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.907103 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:05.907800 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:05.907800 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:05.907801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:05.907806 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:05.909182 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:05.909502 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:05.909503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:05.909503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:05.909555 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:05.909772 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:05.909773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:05.909773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:05.909946 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:05.910024 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:05.910186 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 19:00:05.910281 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:00:05.910370 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:05.905210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:45913)
0925 19:00:05.905349 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:05.905358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:05.905427 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:05.905556 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:05.905557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:05.905745 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:05.905794 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:05.905796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.906373 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.906373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:05.906797 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:05.906798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:05.907115 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:05.907116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:05.907168 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:05.907485 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:05.907489 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:05.907732 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:05.909256 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:05.909256 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:05.909257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:05.909417 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:05.909644 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:05.909644 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:05.909645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:05.909711 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:05.910019 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:05.910020 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:05.910077 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 19:00:05.910161 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:00:05.910253 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:00:05.932969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36317
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:06.008940 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:06.009164 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:06.009212 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:06.010308 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:06.010370 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:06.010403 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 19:00:06.010433 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:00:06.010471 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:06.011812 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:06.012048 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:06.012266 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:06.016933 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:06.016953 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:06.016964 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:06.017069 2474 server_base.cc:1034] running on GCE node
I20260925 19:00:06.017319 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:06.017567 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:06.018721 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790362806018710 us; error 43 us; skew 500 ppm
Sep 25 19:00:06 dist-test-slave-49bz krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:06.021093 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:00:06.021503 2474 webserver.cc:460] Webserver started at http://127.0.98.3:42455/ using document root <none> and password file <none>
I20260925 19:00:06.021726 2474 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:06.021792 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:06.021938 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:06.022941 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "9cc1738d4efa41c4b6344d164c27a90e"
format_stamp: "Formatted at 2026-09-25 19:00:06 on dist-test-slave-49bz"
server_key: "8a1f299c315e6901f1e6458017afbdea"
server_key_iv: "334500c0015e439b772c6344d5088441"
server_key_version: "encryptionkey@0"
I20260925 19:00:06.023277 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9cc1738d4efa41c4b6344d164c27a90e"
format_stamp: "Formatted at 2026-09-25 19:00:06 on dist-test-slave-49bz"
server_key: "8a1f299c315e6901f1e6458017afbdea"
server_key_iv: "334500c0015e439b772c6344d5088441"
server_key_version: "encryptionkey@0"
I20260925 19:00:06.024487 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:00:06.025213 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:06.025396 2474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:06.025482 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "9cc1738d4efa41c4b6344d164c27a90e"
format_stamp: "Formatted at 2026-09-25 19:00:06 on dist-test-slave-49bz"
server_key: "8a1f299c315e6901f1e6458017afbdea"
server_key_iv: "334500c0015e439b772c6344d5088441"
server_key_version: "encryptionkey@0"
I20260925 19:00:06.025563 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:06.043162 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:06.045306 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:06.045463 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:06.045908 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:06.045976 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:06.046010 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:06.046054 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:00:06 dist-test-slave-49bz krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:06.052984 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35667
I20260925 19:00:06.053092 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35667 every 8 connection(s)
I20260925 19:00:06.053354 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:00:06.056079 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.046354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56301 (local address 127.0.98.62:36317)
0925 19:00:06.046417 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.046420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.046723 (+ 303us) server_negotiation.cc:407] Connection header received
0925 19:00:06.047380 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.047381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.047390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.047449 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:06.048302 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.049100 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.050744 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.050946 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.051033 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.053467 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:06.053469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:06.053469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:06.053477 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:06.054813 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:06.055220 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:06.055222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:06.055222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:06.055277 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:06.055514 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:06.055515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:06.055515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:06.055617 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:06.055680 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.055920 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.055998 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:00:06.056236 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.046398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:56301)
0925 19:00:06.046585 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.046628 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.047265 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.047521 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.047526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.047900 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:06.048200 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.048208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.049211 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.049212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.050638 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.050641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.051054 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.051055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.051207 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.051638 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:06.051648 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:06.053375 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:06.054911 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:06.054917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:06.054919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:06.055123 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:06.055337 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:06.055337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:06.055338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:06.055438 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:06.055684 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:06.055689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:06.055807 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.055901 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.056036 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260925 19:00:06.056679 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36317
I20260925 19:00:06.056754 2605 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:06.056937 2605 heartbeater.cc:511] Master 127.0.98.62:36317 requested a full tablet report, sending...
I20260925 19:00:06.057003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 2474
I20260925 19:00:06.057090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:00:06.057281 2121 ts_manager.cc:194] Registered new tserver with Master: 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667)
I20260925 19:00:06.057317 392 external_mini_cluster.cc:1428] Setting key a03503b61b74432bdbcc6faa3d8597c0
I20260925 19:00:06.057852 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56301
I20260925 19:00:06.058284 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:00:06.064371 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.057714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:39841)
0925 19:00:06.057892 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.057903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.057954 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.058699 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.058700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.058836 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:06.058884 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.058885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.059781 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.059782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.060539 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.060541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.060876 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.060877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.060950 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.061329 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:06.061334 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:06.061725 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:06.063495 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:06.063495 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:06.063496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:06.063642 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:06.063836 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:06.063836 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:06.063836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:06.063881 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:06.064139 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:06.064139 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:06.064170 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.064217 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.064280 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260925 19:00:06.064428 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.058023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39841 (local address 127.0.98.62:36317)
0925 19:00:06.058055 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.058056 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.058062 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:00:06.058075 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.058076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.058083 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.058120 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:06.058988 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.059691 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.060618 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.060804 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.060854 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.061801 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:06.061816 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:06.061817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:06.061824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:06.063400 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:06.063714 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:06.063715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:06.063716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:06.063771 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:06.063951 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:06.063952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:06.063953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:06.064080 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:06.064144 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.064241 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.064313 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 25 19:00:06 dist-test-slave-49bz krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:00:06 dist-test-slave-49bz krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:06.080283 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.074229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.0.98.62:36317)
0925 19:00:06.074319 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.074321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.074335 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:00:06.074388 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.074389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.074396 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.074451 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:06.074918 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.075362 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.075999 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.076158 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.076204 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.078230 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:06.078231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:06.078231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:06.078236 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:06.079472 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:06.079765 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:06.079765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:06.079766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:06.079811 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:06.079949 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:06.079949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:06.079949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:06.080035 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:06.080112 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.080186 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.080231 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 19:00:06.081718 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:00:06.083343 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:06.088896 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.085255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35667 (local address 127.0.0.1:60418)
0925 19:00:06.085451 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.085460 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.085513 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.085816 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.085817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.085820 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:06.085861 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.085862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.087048 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.087048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.087850 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.087852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.088702 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.088703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.088754 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.088764 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.088812 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.088847 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":9,"thread_start_us":54,"threads_started":1}
I20260925 19:00:06.088950 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.085526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.0.98.3:35667)
0925 19:00:06.085658 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.085681 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.085690 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:06.085718 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.085719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.085732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.085813 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:06.085925 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.086953 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.087970 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.088624 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.088697 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.088748 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.088831 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.088864 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260925 19:00:06.089521 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.085435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40503 (local address 127.0.0.1:45140)
0925 19:00:06.085894 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.085904 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.085947 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.086217 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.086218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.086219 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:06.086279 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.086281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.087090 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.087091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.088333 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.088335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.089281 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.089282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.089356 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.089373 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.089419 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.089459 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":66,"thread_start_us":32,"threads_started":1}
I20260925 19:00:06.089805 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.085754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.0.98.1:40503)
0925 19:00:06.086020 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.086025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.086034 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:06.086068 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.086070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.086085 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.086151 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:06.086337 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.087025 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.088421 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.089156 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.089270 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.089321 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.089678 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.089716 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":71,"threads_started":1}
I20260925 19:00:06.089867 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 895b694975c34cd1a8434d3439e118db (DEFAULT_TABLE table=test-table [id=b6ab3c7ae26a457f9d5cb7c7ce07f320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:06.090219 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895b694975c34cd1a8434d3439e118db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:06.090399 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.085214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42591 (local address 127.0.0.1:52948)
0925 19:00:06.085393 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.085428 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.085513 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.085708 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.085712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.085713 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:06.085778 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.085780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.087078 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.087079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.088333 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.088335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.090164 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.090165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.090221 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.090224 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.090285 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.090323 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":84,"thread_start_us":80,"threads_started":1}
I20260925 19:00:06.090384 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 895b694975c34cd1a8434d3439e118db (DEFAULT_TABLE table=test-table [id=b6ab3c7ae26a457f9d5cb7c7ce07f320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:06.090649 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.085308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.0.98.2:42591)
0925 19:00:06.085465 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.085470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.085500 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:00:06.085590 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.085592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.085609 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.085691 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:06.085858 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.086990 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.089356 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.090090 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.090153 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.090513 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.090537 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.090572 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260925 19:00:06.090665 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895b694975c34cd1a8434d3439e118db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:06.091118 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 895b694975c34cd1a8434d3439e118db (DEFAULT_TABLE table=test-table [id=b6ab3c7ae26a457f9d5cb7c7ce07f320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:06.091331 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895b694975c34cd1a8434d3439e118db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:06.093405 2625 tablet_bootstrap.cc:492] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: Bootstrap starting.
I20260925 19:00:06.093940 2626 tablet_bootstrap.cc:492] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: Bootstrap starting.
I20260925 19:00:06.094287 2627 tablet_bootstrap.cc:492] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: Bootstrap starting.
I20260925 19:00:06.094326 2625 tablet_bootstrap.cc:654] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:06.094573 2626 tablet_bootstrap.cc:654] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:06.094652 2625 log.cc:824] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:06.094882 2627 tablet_bootstrap.cc:654] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:06.094923 2626 log.cc:824] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:06.095211 2627 log.cc:824] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:06.095381 2625 tablet_bootstrap.cc:492] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: No bootstrap required, opened a new log
I20260925 19:00:06.095475 2625 ts_tablet_manager.cc:1397] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:00:06.095695 2626 tablet_bootstrap.cc:492] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: No bootstrap required, opened a new log
I20260925 19:00:06.095777 2626 ts_tablet_manager.cc:1397] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:00:06.095892 2627 tablet_bootstrap.cc:492] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: No bootstrap required, opened a new log
I20260925 19:00:06.095963 2627 ts_tablet_manager.cc:1397] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:06.097150 2626 raft_consensus.cc:348] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.097296 2626 raft_consensus.cc:374] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:06.097354 2626 raft_consensus.cc:729] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f0cdf4ac7e43138e291e2e0a70bc93, State: Initialized, Role: FOLLOWER
I20260925 19:00:06.097497 2626 consensus_queue.cc:260] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.097455 2625 raft_consensus.cc:348] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.097596 2625 raft_consensus.cc:374] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:06.097667 2625 raft_consensus.cc:729] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc1738d4efa41c4b6344d164c27a90e, State: Initialized, Role: FOLLOWER
I20260925 19:00:06.097790 2626 ts_tablet_manager.cc:1428] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:00:06.097806 2625 consensus_queue.cc:260] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.097867 2325 heartbeater.cc:503] Master 127.0.98.62:36317 was elected leader, sending a full tablet report...
I20260925 19:00:06.098120 2625 ts_tablet_manager.cc:1428] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 19:00:06.098162 2605 heartbeater.cc:503] Master 127.0.98.62:36317 was elected leader, sending a full tablet report...
I20260925 19:00:06.098073 2627 raft_consensus.cc:348] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.098212 2627 raft_consensus.cc:374] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:06.098245 2627 raft_consensus.cc:729] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589e62e73d3e42d4ac54b4d884a29c69, State: Initialized, Role: FOLLOWER
I20260925 19:00:06.098385 2627 consensus_queue.cc:260] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.098642 2627 ts_tablet_manager.cc:1428] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:00:06.098672 2465 heartbeater.cc:503] Master 127.0.98.62:36317 was elected leader, sending a full tablet report...
W20260925 19:00:06.151357 2466 tablet.cc:2369] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:00:06.254694 2326 tablet.cc:2369] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:00:06.303983 2606 tablet.cc:2369] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:00:06.315106 2632 raft_consensus.cc:482] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:06.315201 2632 raft_consensus.cc:504] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.315480 2632 leader_election.cc:288] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591), 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1:40503)
I20260925 19:00:06.318199 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.315654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42591 (local address 127.0.98.3:38801)
0925 19:00:06.315710 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.315723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.315792 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.315972 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.315972 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.315974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:06.316027 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.316029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.316659 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.316660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.317358 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.317360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.317917 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.317918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.317995 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.317999 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.318090 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.318130 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:00:06.318198 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.315755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38801 (local address 127.0.98.2:42591)
0925 19:00:06.315803 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.315806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.315817 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:00:06.315849 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.315850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.315862 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.315917 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:06.316108 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.316562 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.317468 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.317840 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.317928 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.317938 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.318100 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.318135 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:00:06.318351 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.315757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40503 (local address 127.0.98.3:44605)
0925 19:00:06.315931 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.315941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.316021 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.316164 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.316164 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.316166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:06.316242 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.316243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.316833 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.316834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.317527 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.317528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.318061 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.318062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.318133 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.318142 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.318217 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.318272 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260925 19:00:06.318342 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.315846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44605 (local address 127.0.98.1:40503)
0925 19:00:06.315892 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.315894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.316008 (+ 114us) server_negotiation.cc:407] Connection header received
0925 19:00:06.316072 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.316072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.316084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.316146 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:06.316305 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.316746 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.317622 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.317963 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.318060 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.318069 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.318226 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.318282 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:00:06.318756 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895b694975c34cd1a8434d3439e118db" candidate_uuid: "9cc1738d4efa41c4b6344d164c27a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" is_pre_election: true
I20260925 19:00:06.318766 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895b694975c34cd1a8434d3439e118db" candidate_uuid: "9cc1738d4efa41c4b6344d164c27a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589e62e73d3e42d4ac54b4d884a29c69" is_pre_election: true
I20260925 19:00:06.318929 2279 raft_consensus.cc:2454] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cc1738d4efa41c4b6344d164c27a90e in term 0.
I20260925 19:00:06.318933 2419 raft_consensus.cc:2454] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cc1738d4efa41c4b6344d164c27a90e in term 0.
I20260925 19:00:06.319164 2490 leader_election.cc:302] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f0cdf4ac7e43138e291e2e0a70bc93, 9cc1738d4efa41c4b6344d164c27a90e; no voters:
I20260925 19:00:06.319284 2632 raft_consensus.cc:2781] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:06.319347 2632 raft_consensus.cc:482] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:06.319375 2632 raft_consensus.cc:3037] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:06.320012 2632 raft_consensus.cc:504] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.320148 2632 leader_election.cc:288] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [CANDIDATE]: Term 1 election: Requested vote from peers 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591), 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1:40503)
I20260925 19:00:06.320294 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895b694975c34cd1a8434d3439e118db" candidate_uuid: "9cc1738d4efa41c4b6344d164c27a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589e62e73d3e42d4ac54b4d884a29c69"
I20260925 19:00:06.320335 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895b694975c34cd1a8434d3439e118db" candidate_uuid: "9cc1738d4efa41c4b6344d164c27a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93"
I20260925 19:00:06.320389 2419 raft_consensus.cc:3037] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:06.320417 2279 raft_consensus.cc:3037] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:06.321040 2419 raft_consensus.cc:2454] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cc1738d4efa41c4b6344d164c27a90e in term 1.
I20260925 19:00:06.321075 2279 raft_consensus.cc:2454] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cc1738d4efa41c4b6344d164c27a90e in term 1.
I20260925 19:00:06.321234 2490 leader_election.cc:302] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f0cdf4ac7e43138e291e2e0a70bc93, 9cc1738d4efa41c4b6344d164c27a90e; no voters:
I20260925 19:00:06.321341 2632 raft_consensus.cc:2781] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:06.321499 2632 raft_consensus.cc:686] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 1 LEADER]: Becoming Leader. State: Replica: 9cc1738d4efa41c4b6344d164c27a90e, State: Running, Role: LEADER
I20260925 19:00:06.321592 2632 consensus_queue.cc:237] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.322371 2121 catalog_manager.cc:5495] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9cc1738d4efa41c4b6344d164c27a90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } health_report { overall_health: HEALTHY } } }
Sep 25 19:00:06 dist-test-slave-49bz krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:06.393306 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.382842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.0.1:55646)
0925 19:00:06.382974 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.382985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.383049 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.383176 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.383176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.383458 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:06.383508 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.383510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.384163 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.384163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.384683 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.384684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.385002 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.385003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.385039 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.389925 (+ 4886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:06.389930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:06.391207 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:06.392393 (+ 1186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:06.392399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:06.392401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:06.392590 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:06.392757 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:06.392757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:06.392757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:06.392808 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:06.393005 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:06.393011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:06.393049 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.393100 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.393186 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260925 19:00:06.393352 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.382907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.0.98.62:36317)
0925 19:00:06.382972 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.382974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.383048 (+ 74us) server_negotiation.cc:407] Connection header received
0925 19:00:06.383100 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.383101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.383109 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.383155 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:06.383600 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.384094 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.384752 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.384934 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.384989 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.391279 (+ 6290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:06.391280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:06.391281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:06.391286 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:06.392313 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:06.392650 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:06.392650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:06.392651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:06.392694 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:06.392855 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:06.392856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:06.392856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:06.392947 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:06.392992 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.393131 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.393231 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":42}
I20260925 19:00:06.395184 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55646:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 19:00:06.395376 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:06.396828 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 4c069947e61f4cc588ffef393b618474 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0d80bb5808f451489a2f8725413d30b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:00:06.396860 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 4c069947e61f4cc588ffef393b618474 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0d80bb5808f451489a2f8725413d30b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:00:06.397022 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c069947e61f4cc588ffef393b618474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:06.397032 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c069947e61f4cc588ffef393b618474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:06.397055 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 4c069947e61f4cc588ffef393b618474 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0d80bb5808f451489a2f8725413d30b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:00:06.397181 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c069947e61f4cc588ffef393b618474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:06.398165 2627 tablet_bootstrap.cc:492] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: Bootstrap starting.
I20260925 19:00:06.398569 2626 tablet_bootstrap.cc:492] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: Bootstrap starting.
I20260925 19:00:06.398622 2625 tablet_bootstrap.cc:492] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: Bootstrap starting.
I20260925 19:00:06.398890 2627 tablet_bootstrap.cc:654] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:06.398988 2626 tablet_bootstrap.cc:654] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:06.399024 2625 tablet_bootstrap.cc:654] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:06.399556 2626 tablet_bootstrap.cc:492] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: No bootstrap required, opened a new log
I20260925 19:00:06.399581 2627 tablet_bootstrap.cc:492] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: No bootstrap required, opened a new log
I20260925 19:00:06.399622 2626 ts_tablet_manager.cc:1397] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 19:00:06.399618 2627 ts_tablet_manager.cc:1397] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:06.399782 2626 raft_consensus.cc:348] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.399791 2627 raft_consensus.cc:348] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.399858 2626 raft_consensus.cc:374] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:06.399883 2626 raft_consensus.cc:729] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f0cdf4ac7e43138e291e2e0a70bc93, State: Initialized, Role: FOLLOWER
I20260925 19:00:06.399865 2627 raft_consensus.cc:374] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:06.399909 2627 raft_consensus.cc:729] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589e62e73d3e42d4ac54b4d884a29c69, State: Initialized, Role: FOLLOWER
I20260925 19:00:06.399943 2626 consensus_queue.cc:260] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.399966 2627 consensus_queue.cc:260] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.400050 2627 ts_tablet_manager.cc:1428] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:06.400063 2626 ts_tablet_manager.cc:1428] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:06.400244 2631 tablet_replica.cc:439] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } } }
I20260925 19:00:06.400250 2633 tablet_replica.cc:439] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } } }
I20260925 19:00:06.400326 2631 tablet_replica.cc:442] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:06.400344 2633 tablet_replica.cc:442] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:06.400852 2625 tablet_bootstrap.cc:492] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: No bootstrap required, opened a new log
I20260925 19:00:06.400908 2625 ts_tablet_manager.cc:1397] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:00:06.401046 2625 raft_consensus.cc:348] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.401115 2625 raft_consensus.cc:374] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:06.401130 2625 raft_consensus.cc:729] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc1738d4efa41c4b6344d164c27a90e, State: Initialized, Role: FOLLOWER
I20260925 19:00:06.401191 2625 consensus_queue.cc:260] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.401284 2625 ts_tablet_manager.cc:1428] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:06.401317 2632 tablet_replica.cc:439] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } } }
I20260925 19:00:06.401383 2632 tablet_replica.cc:442] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:06.661967 2631 raft_consensus.cc:482] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:06.662099 2631 raft_consensus.cc:504] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.662529 2631 leader_election.cc:288] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591), 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667)
I20260925 19:00:06.665477 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.662764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60755 (local address 127.0.98.2:42591)
0925 19:00:06.662844 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.662846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.662909 (+ 63us) server_negotiation.cc:407] Connection header received
0925 19:00:06.662966 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.662967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.662998 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.663051 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:06.663219 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.663678 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.664682 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.665179 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.665227 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.665235 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.665401 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.665427 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 19:00:06.665545 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.662717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42591 (local address 127.0.98.1:60755)
0925 19:00:06.662826 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.662838 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.662910 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.663054 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.663055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.663058 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:06.663146 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.663148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.663759 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.663760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.664589 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.664591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.665261 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.665263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.665327 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.665331 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.665387 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.665424 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":33,"thread_start_us":54,"threads_started":1}
I20260925 19:00:06.665797 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c069947e61f4cc588ffef393b618474" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589e62e73d3e42d4ac54b4d884a29c69" is_pre_election: true
I20260925 19:00:06.665892 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.662732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35667 (local address 127.0.98.1:44801)
0925 19:00:06.662832 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.662843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.662900 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.663062 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.663063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.663065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:06.663145 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.663148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.663805 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.663806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.664983 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.664986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.665664 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.665665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.665733 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.665736 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.665784 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.665821 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":31,"thread_start_us":48,"threads_started":1}
I20260925 19:00:06.665889 2419 raft_consensus.cc:2454] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 0.
I20260925 19:00:06.666050 2210 leader_election.cc:302] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f0cdf4ac7e43138e291e2e0a70bc93, 589e62e73d3e42d4ac54b4d884a29c69; no voters:
I20260925 19:00:06.666098 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.662784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44801 (local address 127.0.98.3:35667)
0925 19:00:06.662879 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.662881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.662890 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:06.662975 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.662976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.662987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.663058 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:06.663241 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.663700 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.665068 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.665558 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.665623 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.665634 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.665808 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.665857 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":44,"threads_started":1}
I20260925 19:00:06.666179 2631 raft_consensus.cc:2781] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:06.666227 2631 raft_consensus.cc:482] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:06.666275 2631 raft_consensus.cc:3037] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:06.666373 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c069947e61f4cc588ffef393b618474" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc1738d4efa41c4b6344d164c27a90e" is_pre_election: true
I20260925 19:00:06.666466 2559 raft_consensus.cc:2454] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 0.
I20260925 19:00:06.667062 2631 raft_consensus.cc:504] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.667196 2631 leader_election.cc:288] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 election: Requested vote from peers 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591), 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667)
I20260925 19:00:06.667304 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c069947e61f4cc588ffef393b618474" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589e62e73d3e42d4ac54b4d884a29c69"
I20260925 19:00:06.667338 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c069947e61f4cc588ffef393b618474" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc1738d4efa41c4b6344d164c27a90e"
I20260925 19:00:06.667387 2419 raft_consensus.cc:3037] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:06.667392 2559 raft_consensus.cc:3037] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:06.668164 2419 raft_consensus.cc:2454] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 1.
I20260925 19:00:06.668205 2559 raft_consensus.cc:2454] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 1.
I20260925 19:00:06.668341 2210 leader_election.cc:302] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f0cdf4ac7e43138e291e2e0a70bc93, 589e62e73d3e42d4ac54b4d884a29c69; no voters:
I20260925 19:00:06.668468 2631 raft_consensus.cc:2781] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:06.668599 2631 raft_consensus.cc:686] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 LEADER]: Becoming Leader. State: Replica: 21f0cdf4ac7e43138e291e2e0a70bc93, State: Running, Role: LEADER
I20260925 19:00:06.668695 2631 consensus_queue.cc:237] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.669094 2649 tablet_replica.cc:439] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: TxnStatusTablet state changed. Reason: New leader 21f0cdf4ac7e43138e291e2e0a70bc93. Latest consensus state: current_term: 1 leader_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } } }
I20260925 19:00:06.669234 2649 tablet_replica.cc:442] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:00:06.669467 2122 catalog_manager.cc:5495] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } health_report { overall_health: UNKNOWN } } }
I20260925 19:00:06.669766 2651 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 19:00:06.669884 2651 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 19:00:06.682327 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.679770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45150 (local address 127.0.98.1:40503)
0925 19:00:06.679831 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.679833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.679842 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:06.679869 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.679870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.679888 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.679962 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:06.680072 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.680792 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.681508 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.681788 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.681848 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.682002 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:06.682106 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:06.682181 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.682205 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.682223 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:00:06.682341 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.679571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40503 (local address 127.0.0.1:45150)
0925 19:00:06.679647 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.679666 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.679706 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.679972 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.679973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.679974 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:06.680012 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.680013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.680862 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.680863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.681416 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.681418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.681863 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.681864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.681908 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.681920 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:06.682167 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:06.682173 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.682199 (+ 26us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.682233 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 19:00:06.683939 2419 raft_consensus.cc:1261] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Refusing update from remote peer 21f0cdf4ac7e43138e291e2e0a70bc93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:06.683959 2559 raft_consensus.cc:1261] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Refusing update from remote peer 21f0cdf4ac7e43138e291e2e0a70bc93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:06.684258 2631 consensus_queue.cc:1035] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:06.684386 2649 consensus_queue.cc:1035] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:06.685393 2632 tablet_replica.cc:439] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: TxnStatusTablet state changed. Reason: New leader 21f0cdf4ac7e43138e291e2e0a70bc93. Latest consensus state: current_term: 1 leader_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } } }
I20260925 19:00:06.685496 2632 tablet_replica.cc:442] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:06.685554 2633 tablet_replica.cc:439] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: TxnStatusTablet state changed. Reason: New leader 21f0cdf4ac7e43138e291e2e0a70bc93. Latest consensus state: current_term: 1 leader_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } } }
I20260925 19:00:06.685629 2633 tablet_replica.cc:442] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:06.686094 2649 tablet_replica.cc:439] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } } }
I20260925 19:00:06.686170 2649 tablet_replica.cc:442] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:00:06.686235 2654 mvcc.cc:205] Tried to move back new op lower bound from 7333326056174993408 to 7333326056115384320. Current Snapshot: MvccSnapshot[applied={T|T < 7333326056174993408}]
I20260925 19:00:06.686483 2632 tablet_replica.cc:439] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } } }
I20260925 19:00:06.686558 2632 tablet_replica.cc:442] T 4c069947e61f4cc588ffef393b618474 P 9cc1738d4efa41c4b6344d164c27a90e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:06.686895 2656 mvcc.cc:205] Tried to move back new op lower bound from 7333326056174993408 to 7333326056115384320. Current Snapshot: MvccSnapshot[applied={T|T < 7333326056174993408 or (T in {7333326056174993408})}]
I20260925 19:00:06.687188 2649 tablet_replica.cc:439] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } } }
I20260925 19:00:06.687266 2649 tablet_replica.cc:442] T 4c069947e61f4cc588ffef393b618474 P 21f0cdf4ac7e43138e291e2e0a70bc93: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:00:06.687618 2655 mvcc.cc:205] Tried to move back new op lower bound from 7333326056174993408 to 7333326056115384320. Current Snapshot: MvccSnapshot[applied={T|T < 7333326056174993408}]
I20260925 19:00:06.687687 2633 tablet_replica.cc:439] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } } }
I20260925 19:00:06.687764 2633 tablet_replica.cc:442] T 4c069947e61f4cc588ffef393b618474 P 589e62e73d3e42d4ac54b4d884a29c69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:06.691121 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.688741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.0.98.3:35667)
0925 19:00:06.688783 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.688785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.688864 (+ 79us) server_negotiation.cc:407] Connection header received
0925 19:00:06.688944 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.688945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.688961 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.689011 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:06.689140 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.689603 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.690360 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.690567 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.690641 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.690805 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:06.690894 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:06.690982 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.691019 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.691037 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:00:06.699156 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.693263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53987 (local address 127.0.98.1:40503)
0925 19:00:06.693317 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.693319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.696774 (+ 3455us) server_negotiation.cc:407] Connection header received
0925 19:00:06.696797 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.696798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.696808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.696857 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:06.696972 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.697648 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.698432 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.698729 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.698798 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.698946 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:06.698978 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:06.699030 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.699078 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.699094 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:00:06.699163 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.693216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40503 (local address 127.0.98.3:53987)
0925 19:00:06.696674 (+ 3458us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.696684 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.696734 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.696866 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.696866 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.696867 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:06.696909 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.696910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.697732 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.697733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.698362 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.698364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.698802 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.698803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.698845 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.698856 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:06.699038 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:06.699040 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.699074 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.699106 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3430,"thread_start_us":76,"threads_started":1}
I20260925 19:00:06.701519 2419 raft_consensus.cc:1261] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Refusing update from remote peer 9cc1738d4efa41c4b6344d164c27a90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:06.701713 2279 raft_consensus.cc:1261] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Refusing update from remote peer 9cc1738d4efa41c4b6344d164c27a90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:06.701901 2635 consensus_queue.cc:1035] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:06.702013 2632 consensus_queue.cc:1035] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:06.709656 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.706801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35667 (local address 127.0.98.1:60215)
0925 19:00:06.706948 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.706959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.707003 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.707099 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.707099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.707100 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:06.707139 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.707140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.707941 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.707942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.708819 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.708821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.709287 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.709288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.709341 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.709352 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:06.709529 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:06.709531 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.709565 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.709595 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260925 19:00:06.709651 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.706971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60215 (local address 127.0.98.3:35667)
0925 19:00:06.707009 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.707011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.707019 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:06.707040 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.707041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.707048 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.707098 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:06.707173 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.707862 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.708888 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.709210 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.709280 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.709433 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:06.709462 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:06.709519 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.709567 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.709586 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:06.719123 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:06.719221 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:06.719959 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b: Sending DeleteTablet for 3 replicas of tablet 895b694975c34cd1a8434d3439e118db
I20260925 19:00:06.720314 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 895b694975c34cd1a8434d3439e118db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60418
I20260925 19:00:06.720327 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 895b694975c34cd1a8434d3439e118db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52948
I20260925 19:00:06.720500 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 895b694975c34cd1a8434d3439e118db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45140
I20260925 19:00:06.720497 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:00:06.720639 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:06.721769 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 62e1136dde1e4c35b8104612b5d6b546 (DEFAULT_TABLE table=test-table [id=1a2581e17786436c956d39e45b787163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:06.721760 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 62e1136dde1e4c35b8104612b5d6b546 (DEFAULT_TABLE table=test-table [id=1a2581e17786436c956d39e45b787163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:06.721807 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 62e1136dde1e4c35b8104612b5d6b546 (DEFAULT_TABLE table=test-table [id=1a2581e17786436c956d39e45b787163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:06.721866 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e1136dde1e4c35b8104612b5d6b546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:06.721864 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e1136dde1e4c35b8104612b5d6b546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:06.721911 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e1136dde1e4c35b8104612b5d6b546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:06.723062 2625 tablet_bootstrap.cc:492] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e: Bootstrap starting.
I20260925 19:00:06.723096 2626 tablet_bootstrap.cc:492] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93: Bootstrap starting.
I20260925 19:00:06.723165 2627 tablet_bootstrap.cc:492] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69: Bootstrap starting.
I20260925 19:00:06.723624 2627 tablet_bootstrap.cc:654] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:06.723869 2625 tablet_bootstrap.cc:654] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:06.723904 2626 tablet_bootstrap.cc:654] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:06.724248 2627 tablet_bootstrap.cc:492] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69: No bootstrap required, opened a new log
I20260925 19:00:06.724295 2627 ts_tablet_manager.cc:1397] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:06.724442 2627 raft_consensus.cc:348] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.724529 2627 raft_consensus.cc:374] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:06.724556 2627 raft_consensus.cc:729] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589e62e73d3e42d4ac54b4d884a29c69, State: Initialized, Role: FOLLOWER
I20260925 19:00:06.724603 2627 consensus_queue.cc:260] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.724711 2627 ts_tablet_manager.cc:1428] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:06.724797 2625 tablet_bootstrap.cc:492] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e: No bootstrap required, opened a new log
I20260925 19:00:06.724843 2625 ts_tablet_manager.cc:1397] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:00:06.725001 2625 raft_consensus.cc:348] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.725101 2625 raft_consensus.cc:374] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:06.725126 2625 raft_consensus.cc:729] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc1738d4efa41c4b6344d164c27a90e, State: Initialized, Role: FOLLOWER
I20260925 19:00:06.725169 2625 consensus_queue.cc:260] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.725253 2625 ts_tablet_manager.cc:1428] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:06.730556 2678 tablet_replica.cc:330] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: stopping tablet replica
I20260925 19:00:06.730664 2626 tablet_bootstrap.cc:492] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93: No bootstrap required, opened a new log
I20260925 19:00:06.730728 2626 ts_tablet_manager.cc:1397] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260925 19:00:06.730858 2626 raft_consensus.cc:348] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.730931 2626 raft_consensus.cc:374] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:06.730962 2626 raft_consensus.cc:729] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f0cdf4ac7e43138e291e2e0a70bc93, State: Initialized, Role: FOLLOWER
I20260925 19:00:06.731003 2626 consensus_queue.cc:260] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.731105 2626 ts_tablet_manager.cc:1428] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:06.731323 2677 tablet_replica.cc:330] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: stopping tablet replica
I20260925 19:00:06.731412 2677 raft_consensus.cc:2229] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:06.731500 2677 raft_consensus.cc:2258] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:06.731642 2676 tablet_replica.cc:330] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: stopping tablet replica
I20260925 19:00:06.731727 2676 raft_consensus.cc:2229] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:06.731792 2676 raft_consensus.cc:2258] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:06.730667 2678 raft_consensus.cc:2229] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:06.731940 2678 raft_consensus.cc:2258] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:06.732383 2678 ts_tablet_manager.cc:1905] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:06.732543 2676 ts_tablet_manager.cc:1905] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:06.733538 2677 ts_tablet_manager.cc:1905] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:06.734301 2678 ts_tablet_manager.cc:1918] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:06.734417 2678 log.cc:1196] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/895b694975c34cd1a8434d3439e118db
I20260925 19:00:06.735199 2676 ts_tablet_manager.cc:1918] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:06.735273 2676 log.cc:1196] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/895b694975c34cd1a8434d3439e118db
I20260925 19:00:06.735536 2676 ts_tablet_manager.cc:1939] T 895b694975c34cd1a8434d3439e118db P 589e62e73d3e42d4ac54b4d884a29c69: Deleting consensus metadata
I20260925 19:00:06.735639 2678 ts_tablet_manager.cc:1939] T 895b694975c34cd1a8434d3439e118db P 21f0cdf4ac7e43138e291e2e0a70bc93: Deleting consensus metadata
I20260925 19:00:06.735919 2106 catalog_manager.cc:4841] TS 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591): tablet 895b694975c34cd1a8434d3439e118db (table test-table [id=b6ab3c7ae26a457f9d5cb7c7ce07f320]) successfully deleted
I20260925 19:00:06.736027 2106 catalog_manager.cc:4841] TS 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1:40503): tablet 895b694975c34cd1a8434d3439e118db (table test-table [id=b6ab3c7ae26a457f9d5cb7c7ce07f320]) successfully deleted
I20260925 19:00:06.737005 2677 ts_tablet_manager.cc:1918] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:06.737082 2677 log.cc:1196] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/895b694975c34cd1a8434d3439e118db
I20260925 19:00:06.737368 2677 ts_tablet_manager.cc:1939] T 895b694975c34cd1a8434d3439e118db P 9cc1738d4efa41c4b6344d164c27a90e: Deleting consensus metadata
I20260925 19:00:06.737752 2109 catalog_manager.cc:4841] TS 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667): tablet 895b694975c34cd1a8434d3439e118db (table test-table [id=b6ab3c7ae26a457f9d5cb7c7ce07f320]) successfully deleted
I20260925 19:00:06.818626 2667 raft_consensus.cc:482] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:06.818730 2667 raft_consensus.cc:504] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.818881 2667 leader_election.cc:288] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591), 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667)
I20260925 19:00:06.819133 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e1136dde1e4c35b8104612b5d6b546" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589e62e73d3e42d4ac54b4d884a29c69" is_pre_election: true
I20260925 19:00:06.819133 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e1136dde1e4c35b8104612b5d6b546" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc1738d4efa41c4b6344d164c27a90e" is_pre_election: true
I20260925 19:00:06.819239 2419 raft_consensus.cc:2454] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 0.
I20260925 19:00:06.819276 2559 raft_consensus.cc:2454] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 0.
I20260925 19:00:06.819406 2210 leader_election.cc:302] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f0cdf4ac7e43138e291e2e0a70bc93, 589e62e73d3e42d4ac54b4d884a29c69; no voters:
I20260925 19:00:06.819497 2667 raft_consensus.cc:2781] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:06.819531 2667 raft_consensus.cc:482] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:06.819561 2667 raft_consensus.cc:3037] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:06.820400 2667 raft_consensus.cc:504] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.820537 2667 leader_election.cc:288] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 election: Requested vote from peers 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591), 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667)
I20260925 19:00:06.820734 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e1136dde1e4c35b8104612b5d6b546" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc1738d4efa41c4b6344d164c27a90e"
I20260925 19:00:06.820758 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e1136dde1e4c35b8104612b5d6b546" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589e62e73d3e42d4ac54b4d884a29c69"
I20260925 19:00:06.820816 2559 raft_consensus.cc:3037] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:06.820822 2419 raft_consensus.cc:3037] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:06.821612 2559 raft_consensus.cc:2454] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 1.
I20260925 19:00:06.821611 2419 raft_consensus.cc:2454] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 1.
I20260925 19:00:06.821790 2213 leader_election.cc:302] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f0cdf4ac7e43138e291e2e0a70bc93, 9cc1738d4efa41c4b6344d164c27a90e; no voters:
I20260925 19:00:06.821916 2667 raft_consensus.cc:2781] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:06.821982 2667 raft_consensus.cc:686] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 LEADER]: Becoming Leader. State: Replica: 21f0cdf4ac7e43138e291e2e0a70bc93, State: Running, Role: LEADER
I20260925 19:00:06.822027 2667 consensus_queue.cc:237] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:06.822607 2122 catalog_manager.cc:5495] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } health_report { overall_health: UNKNOWN } } }
I20260925 19:00:06.844245 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.841895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.0.98.1:40503)
0925 19:00:06.841949 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.841951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.841958 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:00:06.841977 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.841977 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.841988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.842051 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:06.842197 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.842702 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.843528 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.843713 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.843793 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.844017 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:06.844051 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:06.844098 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.844147 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.844165 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:00:06.848170 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.846162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47343 (local address 127.0.98.1:40503)
0925 19:00:06.846221 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:06.846222 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:06.846229 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:00:06.846273 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:06.846274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:06.846282 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:06.846347 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:06.846455 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.846860 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.847586 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.847749 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.847817 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.847935 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:06.847963 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:06.848011 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:06.848064 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 19:00:06.848077 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:00:06.848170 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:06.846064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40503 (local address 127.0.98.1:47343)
0925 19:00:06.846132 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:06.846143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:06.846203 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:06.846350 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:06.846351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:06.846352 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:06.846384 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.846385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.846947 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.846948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.847497 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:06.847499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:06.847816 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:06.847816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:06.847864 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:06.847868 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:06.848011 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:06.848013 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:00:06.848052 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 19:00:06.848109 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 19:00:06.850003 2559 raft_consensus.cc:1261] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Refusing update from remote peer 21f0cdf4ac7e43138e291e2e0a70bc93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:06.850025 2419 raft_consensus.cc:1261] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Refusing update from remote peer 21f0cdf4ac7e43138e291e2e0a70bc93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:06.850226 2667 consensus_queue.cc:1035] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:06.850344 2631 consensus_queue.cc:1035] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:00:06.854321 2674 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 19:00:08.371599 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:08.371694 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:08.372541 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b: Sending DeleteTablet for 3 replicas of tablet 62e1136dde1e4c35b8104612b5d6b546
I20260925 19:00:08.372783 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e1136dde1e4c35b8104612b5d6b546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52948
I20260925 19:00:08.372784 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e1136dde1e4c35b8104612b5d6b546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45140
I20260925 19:00:08.372988 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e1136dde1e4c35b8104612b5d6b546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60418
I20260925 19:00:08.373001 2707 tablet_replica.cc:330] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93: stopping tablet replica
I20260925 19:00:08.373063 2707 raft_consensus.cc:2229] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:08.373150 2707 raft_consensus.cc:2258] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:08.373159 2708 tablet_replica.cc:330] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e: stopping tablet replica
I20260925 19:00:08.373206 2708 raft_consensus.cc:2229] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:08.373241 2708 raft_consensus.cc:2258] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:08.373404 2707 ts_tablet_manager.cc:1905] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:08.373469 2708 ts_tablet_manager.cc:1905] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:08.375062 2708 ts_tablet_manager.cc:1918] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:00:08.375200 2708 log.cc:1196] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/62e1136dde1e4c35b8104612b5d6b546
I20260925 19:00:08.375293 2707 ts_tablet_manager.cc:1918] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:00:08.375401 2707 log.cc:1196] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/62e1136dde1e4c35b8104612b5d6b546
I20260925 19:00:08.375514 2708 ts_tablet_manager.cc:1939] T 62e1136dde1e4c35b8104612b5d6b546 P 9cc1738d4efa41c4b6344d164c27a90e: Deleting consensus metadata
I20260925 19:00:08.375604 2706 tablet_replica.cc:330] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69: stopping tablet replica
I20260925 19:00:08.375698 2707 ts_tablet_manager.cc:1939] T 62e1136dde1e4c35b8104612b5d6b546 P 21f0cdf4ac7e43138e291e2e0a70bc93: Deleting consensus metadata
I20260925 19:00:08.375720 2706 raft_consensus.cc:2229] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:08.375823 2706 raft_consensus.cc:2258] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:08.375829 2109 catalog_manager.cc:4841] TS 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667): tablet 62e1136dde1e4c35b8104612b5d6b546 (table test-table [id=1a2581e17786436c956d39e45b787163]) successfully deleted
I20260925 19:00:08.376008 2106 catalog_manager.cc:4841] TS 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1:40503): tablet 62e1136dde1e4c35b8104612b5d6b546 (table test-table [id=1a2581e17786436c956d39e45b787163]) successfully deleted
I20260925 19:00:08.376066 2706 ts_tablet_manager.cc:1905] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:08.377243 2706 ts_tablet_manager.cc:1918] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:00:08.377288 2706 log.cc:1196] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/62e1136dde1e4c35b8104612b5d6b546
I20260925 19:00:08.377486 2706 ts_tablet_manager.cc:1939] T 62e1136dde1e4c35b8104612b5d6b546 P 589e62e73d3e42d4ac54b4d884a29c69: Deleting consensus metadata
I20260925 19:00:08.377799 2106 catalog_manager.cc:4841] TS 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591): tablet 62e1136dde1e4c35b8104612b5d6b546 (table test-table [id=1a2581e17786436c956d39e45b787163]) successfully deleted
Sep 25 19:00:08 dist-test-slave-49bz krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:00:08 dist-test-slave-49bz krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:08.394582 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:08.388082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.0.98.62:36317)
0925 19:00:08.388226 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:08.388228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:08.388236 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:08.388254 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:08.388255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:08.388264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:08.388326 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:08.388823 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:08.389318 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:08.390017 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:08.390201 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:08.390244 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:08.392284 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:08.392285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:08.392286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:08.392291 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:08.393709 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:08.393982 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:08.393982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:08.393982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:08.394041 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:08.394176 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:08.394176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:08.394176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:08.394302 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:08.394378 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:08.394462 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 19:00:08.394521 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260925 19:00:08.395853 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:00:08.396000 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:08.397154 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 072b1735ffda4c409e2703b729d7f744 (DEFAULT_TABLE table=test-table [id=114d73e3967e45fcb5a3b39619c58ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:08.397162 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 072b1735ffda4c409e2703b729d7f744 (DEFAULT_TABLE table=test-table [id=114d73e3967e45fcb5a3b39619c58ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:08.397264 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 072b1735ffda4c409e2703b729d7f744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:08.397265 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 072b1735ffda4c409e2703b729d7f744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:08.397243 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 072b1735ffda4c409e2703b729d7f744 (DEFAULT_TABLE table=test-table [id=114d73e3967e45fcb5a3b39619c58ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:08.397344 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 072b1735ffda4c409e2703b729d7f744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:08.398200 2717 tablet_bootstrap.cc:492] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e: Bootstrap starting.
I20260925 19:00:08.398665 2717 tablet_bootstrap.cc:654] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:08.398700 2719 tablet_bootstrap.cc:492] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93: Bootstrap starting.
I20260925 19:00:08.399056 2719 tablet_bootstrap.cc:654] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:08.399281 2717 tablet_bootstrap.cc:492] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e: No bootstrap required, opened a new log
I20260925 19:00:08.399335 2717 ts_tablet_manager.cc:1397] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:08.399365 2718 tablet_bootstrap.cc:492] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69: Bootstrap starting.
I20260925 19:00:08.399498 2717 raft_consensus.cc:348] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.399580 2717 raft_consensus.cc:374] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:08.399603 2717 raft_consensus.cc:729] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc1738d4efa41c4b6344d164c27a90e, State: Initialized, Role: FOLLOWER
I20260925 19:00:08.399676 2719 tablet_bootstrap.cc:492] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93: No bootstrap required, opened a new log
I20260925 19:00:08.399710 2718 tablet_bootstrap.cc:654] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:08.399693 2717 consensus_queue.cc:260] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.399739 2719 ts_tablet_manager.cc:1397] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:08.399883 2717 ts_tablet_manager.cc:1428] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:08.399858 2719 raft_consensus.cc:348] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.399923 2719 raft_consensus.cc:374] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:08.399943 2719 raft_consensus.cc:729] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f0cdf4ac7e43138e291e2e0a70bc93, State: Initialized, Role: FOLLOWER
I20260925 19:00:08.400038 2719 consensus_queue.cc:260] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.400215 2719 ts_tablet_manager.cc:1428] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:08.400630 2718 tablet_bootstrap.cc:492] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69: No bootstrap required, opened a new log
I20260925 19:00:08.400710 2718 ts_tablet_manager.cc:1397] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:08.400843 2718 raft_consensus.cc:348] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.400913 2718 raft_consensus.cc:374] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:08.400941 2718 raft_consensus.cc:729] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589e62e73d3e42d4ac54b4d884a29c69, State: Initialized, Role: FOLLOWER
I20260925 19:00:08.401002 2718 consensus_queue.cc:260] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.401149 2718 ts_tablet_manager.cc:1428] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:08.436053 2722 raft_consensus.cc:482] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:08.436134 2722 raft_consensus.cc:504] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.436288 2722 leader_election.cc:288] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1:40503), 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591)
I20260925 19:00:08.436507 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "072b1735ffda4c409e2703b729d7f744" candidate_uuid: "9cc1738d4efa41c4b6344d164c27a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" is_pre_election: true
I20260925 19:00:08.436524 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "072b1735ffda4c409e2703b729d7f744" candidate_uuid: "9cc1738d4efa41c4b6344d164c27a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589e62e73d3e42d4ac54b4d884a29c69" is_pre_election: true
I20260925 19:00:08.436591 2279 raft_consensus.cc:2454] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cc1738d4efa41c4b6344d164c27a90e in term 0.
I20260925 19:00:08.436621 2419 raft_consensus.cc:2454] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cc1738d4efa41c4b6344d164c27a90e in term 0.
I20260925 19:00:08.436755 2490 leader_election.cc:302] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f0cdf4ac7e43138e291e2e0a70bc93, 9cc1738d4efa41c4b6344d164c27a90e; no voters:
I20260925 19:00:08.436892 2722 raft_consensus.cc:2781] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:08.436931 2722 raft_consensus.cc:482] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:08.436964 2722 raft_consensus.cc:3037] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:08.437572 2722 raft_consensus.cc:504] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.437702 2722 leader_election.cc:288] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [CANDIDATE]: Term 1 election: Requested vote from peers 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1:40503), 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591)
I20260925 19:00:08.437867 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "072b1735ffda4c409e2703b729d7f744" candidate_uuid: "9cc1738d4efa41c4b6344d164c27a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93"
I20260925 19:00:08.437886 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "072b1735ffda4c409e2703b729d7f744" candidate_uuid: "9cc1738d4efa41c4b6344d164c27a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589e62e73d3e42d4ac54b4d884a29c69"
I20260925 19:00:08.437933 2279 raft_consensus.cc:3037] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:08.437953 2419 raft_consensus.cc:3037] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:08.438586 2419 raft_consensus.cc:2454] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cc1738d4efa41c4b6344d164c27a90e in term 1.
I20260925 19:00:08.438591 2279 raft_consensus.cc:2454] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cc1738d4efa41c4b6344d164c27a90e in term 1.
I20260925 19:00:08.438743 2490 leader_election.cc:302] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589e62e73d3e42d4ac54b4d884a29c69, 9cc1738d4efa41c4b6344d164c27a90e; no voters:
I20260925 19:00:08.438833 2722 raft_consensus.cc:2781] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:08.438977 2722 raft_consensus.cc:686] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 LEADER]: Becoming Leader. State: Replica: 9cc1738d4efa41c4b6344d164c27a90e, State: Running, Role: LEADER
I20260925 19:00:08.439023 2722 consensus_queue.cc:237] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.439551 2122 catalog_manager.cc:5495] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9cc1738d4efa41c4b6344d164c27a90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } health_report { overall_health: HEALTHY } } }
I20260925 19:00:08.447600 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:08.445412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.0.98.3:35667)
0925 19:00:08.445554 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:08.445556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:08.445562 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:00:08.445579 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:08.445579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:08.445588 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:08.445640 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:08.445728 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:08.446210 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:08.446932 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:08.447119 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:08.447170 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:08.447354 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:08.447389 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:08.447434 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:08.447490 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 19:00:08.447508 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260925 19:00:08.449530 2279 raft_consensus.cc:1261] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Refusing update from remote peer 9cc1738d4efa41c4b6344d164c27a90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:08.449542 2419 raft_consensus.cc:1261] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Refusing update from remote peer 9cc1738d4efa41c4b6344d164c27a90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:08.449710 2722 consensus_queue.cc:1035] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:08.449831 2725 consensus_queue.cc:1035] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:08.459288 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:08.459357 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:08.459859 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b: Sending DeleteTablet for 3 replicas of tablet 072b1735ffda4c409e2703b729d7f744
I20260925 19:00:08.460069 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 072b1735ffda4c409e2703b729d7f744 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45140
I20260925 19:00:08.460115 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 072b1735ffda4c409e2703b729d7f744 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52948
I20260925 19:00:08.460167 2707 tablet_replica.cc:330] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93: stopping tablet replica
I20260925 19:00:08.460211 2707 raft_consensus.cc:2229] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:08.460199 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 072b1735ffda4c409e2703b729d7f744 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60418
I20260925 19:00:08.460242 2707 raft_consensus.cc:2258] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:08.460269 2706 tablet_replica.cc:330] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69: stopping tablet replica
I20260925 19:00:08.460312 2706 raft_consensus.cc:2229] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:08.460271 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 19:00:08.460368 2706 raft_consensus.cc:2258] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 19:00:08.460402 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:08.460429 2708 tablet_replica.cc:330] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e: stopping tablet replica
I20260925 19:00:08.460489 2708 raft_consensus.cc:2229] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:08.460543 2708 raft_consensus.cc:2258] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:08.460681 2707 ts_tablet_manager.cc:1905] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:08.460925 2706 ts_tablet_manager.cc:1905] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:08.462293 2707 ts_tablet_manager.cc:1918] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:08.462345 2707 log.cc:1196] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/072b1735ffda4c409e2703b729d7f744
I20260925 19:00:08.462416 2706 ts_tablet_manager.cc:1918] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:08.462458 2706 log.cc:1196] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/072b1735ffda4c409e2703b729d7f744
I20260925 19:00:08.462564 2707 ts_tablet_manager.cc:1939] T 072b1735ffda4c409e2703b729d7f744 P 21f0cdf4ac7e43138e291e2e0a70bc93: Deleting consensus metadata
I20260925 19:00:08.462670 2706 ts_tablet_manager.cc:1939] T 072b1735ffda4c409e2703b729d7f744 P 589e62e73d3e42d4ac54b4d884a29c69: Deleting consensus metadata
I20260925 19:00:08.462821 2106 catalog_manager.cc:4841] TS 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1:40503): tablet 072b1735ffda4c409e2703b729d7f744 (table test-table [id=114d73e3967e45fcb5a3b39619c58ebe]) successfully deleted
I20260925 19:00:08.462885 2106 catalog_manager.cc:4841] TS 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591): tablet 072b1735ffda4c409e2703b729d7f744 (table test-table [id=114d73e3967e45fcb5a3b39619c58ebe]) successfully deleted
I20260925 19:00:08.462977 2708 ts_tablet_manager.cc:1905] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:08.464028 2708 ts_tablet_manager.cc:1918] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:08.464075 2708 log.cc:1196] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/072b1735ffda4c409e2703b729d7f744
I20260925 19:00:08.464298 2708 ts_tablet_manager.cc:1939] T 072b1735ffda4c409e2703b729d7f744 P 9cc1738d4efa41c4b6344d164c27a90e: Deleting consensus metadata
I20260925 19:00:08.464463 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 12655b7318714f3a9abd69999031f6aa (DEFAULT_TABLE table=test-table [id=3962770ad7b64ca2a115cb842edcf844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:08.464469 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 12655b7318714f3a9abd69999031f6aa (DEFAULT_TABLE table=test-table [id=3962770ad7b64ca2a115cb842edcf844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:08.464578 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12655b7318714f3a9abd69999031f6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:08.464592 2109 catalog_manager.cc:4841] TS 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667): tablet 072b1735ffda4c409e2703b729d7f744 (table test-table [id=114d73e3967e45fcb5a3b39619c58ebe]) successfully deleted
I20260925 19:00:08.464608 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12655b7318714f3a9abd69999031f6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:08.465479 2717 tablet_bootstrap.cc:492] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e: Bootstrap starting.
I20260925 19:00:08.465493 2718 tablet_bootstrap.cc:492] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69: Bootstrap starting.
I20260925 19:00:08.465669 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 12655b7318714f3a9abd69999031f6aa (DEFAULT_TABLE table=test-table [id=3962770ad7b64ca2a115cb842edcf844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:08.465763 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12655b7318714f3a9abd69999031f6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:08.465795 2717 tablet_bootstrap.cc:654] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:08.466032 2718 tablet_bootstrap.cc:654] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:08.466328 2717 tablet_bootstrap.cc:492] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e: No bootstrap required, opened a new log
I20260925 19:00:08.466382 2717 ts_tablet_manager.cc:1397] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:08.466490 2717 raft_consensus.cc:348] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.466560 2717 raft_consensus.cc:374] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:08.466590 2717 raft_consensus.cc:729] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc1738d4efa41c4b6344d164c27a90e, State: Initialized, Role: FOLLOWER
I20260925 19:00:08.466636 2717 consensus_queue.cc:260] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.466719 2717 ts_tablet_manager.cc:1428] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:08.467015 2719 tablet_bootstrap.cc:492] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93: Bootstrap starting.
I20260925 19:00:08.467286 2719 tablet_bootstrap.cc:654] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:08.467777 2719 tablet_bootstrap.cc:492] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93: No bootstrap required, opened a new log
I20260925 19:00:08.467797 2718 tablet_bootstrap.cc:492] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69: No bootstrap required, opened a new log
I20260925 19:00:08.467828 2719 ts_tablet_manager.cc:1397] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 19:00:08.467832 2718 ts_tablet_manager.cc:1397] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:08.467927 2719 raft_consensus.cc:348] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.467957 2718 raft_consensus.cc:348] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.467993 2719 raft_consensus.cc:374] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:08.468014 2718 raft_consensus.cc:374] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:08.468034 2718 raft_consensus.cc:729] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589e62e73d3e42d4ac54b4d884a29c69, State: Initialized, Role: FOLLOWER
I20260925 19:00:08.468014 2719 raft_consensus.cc:729] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f0cdf4ac7e43138e291e2e0a70bc93, State: Initialized, Role: FOLLOWER
I20260925 19:00:08.468082 2718 consensus_queue.cc:260] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.468087 2719 consensus_queue.cc:260] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.468168 2718 ts_tablet_manager.cc:1428] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:08.468189 2719 ts_tablet_manager.cc:1428] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:08.585561 2688 raft_consensus.cc:482] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:08.585654 2688 raft_consensus.cc:504] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.585811 2688 leader_election.cc:288] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591), 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667)
I20260925 19:00:08.586017 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12655b7318714f3a9abd69999031f6aa" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589e62e73d3e42d4ac54b4d884a29c69" is_pre_election: true
I20260925 19:00:08.586028 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12655b7318714f3a9abd69999031f6aa" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc1738d4efa41c4b6344d164c27a90e" is_pre_election: true
I20260925 19:00:08.586122 2559 raft_consensus.cc:2454] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 0.
I20260925 19:00:08.586123 2419 raft_consensus.cc:2454] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 0.
I20260925 19:00:08.586321 2213 leader_election.cc:302] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f0cdf4ac7e43138e291e2e0a70bc93, 9cc1738d4efa41c4b6344d164c27a90e; no voters:
I20260925 19:00:08.586439 2688 raft_consensus.cc:2781] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:08.586493 2688 raft_consensus.cc:482] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:08.586508 2688 raft_consensus.cc:3037] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:08.587196 2688 raft_consensus.cc:504] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.587327 2688 leader_election.cc:288] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 election: Requested vote from peers 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591), 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667)
I20260925 19:00:08.587468 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12655b7318714f3a9abd69999031f6aa" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589e62e73d3e42d4ac54b4d884a29c69"
I20260925 19:00:08.587496 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12655b7318714f3a9abd69999031f6aa" candidate_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc1738d4efa41c4b6344d164c27a90e"
I20260925 19:00:08.587559 2559 raft_consensus.cc:3037] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:08.587575 2419 raft_consensus.cc:3037] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:08.588088 2559 raft_consensus.cc:2454] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 1.
I20260925 19:00:08.588281 2419 raft_consensus.cc:2454] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f0cdf4ac7e43138e291e2e0a70bc93 in term 1.
I20260925 19:00:08.588289 2213 leader_election.cc:302] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f0cdf4ac7e43138e291e2e0a70bc93, 9cc1738d4efa41c4b6344d164c27a90e; no voters:
I20260925 19:00:08.588399 2688 raft_consensus.cc:2781] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:08.588455 2688 raft_consensus.cc:686] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [term 1 LEADER]: Becoming Leader. State: Replica: 21f0cdf4ac7e43138e291e2e0a70bc93, State: Running, Role: LEADER
I20260925 19:00:08.588510 2688 consensus_queue.cc:237] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } }
I20260925 19:00:08.589067 2122 catalog_manager.cc:5495] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f0cdf4ac7e43138e291e2e0a70bc93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 } health_report { overall_health: UNKNOWN } } }
I20260925 19:00:08.609786 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:08.607679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.0.98.1:40503)
0925 19:00:08.607841 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:08.607843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:08.607851 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:08.607879 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:08.607879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:08.607889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:08.607945 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:08.608065 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:08.608531 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:08.609182 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:08.609359 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:08.609420 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:08.609579 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:08.609603 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:08.609635 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:08.609671 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 19:00:08.609687 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260925 19:00:08.611862 2559 raft_consensus.cc:1261] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Refusing update from remote peer 21f0cdf4ac7e43138e291e2e0a70bc93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:08.611881 2419 raft_consensus.cc:1261] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Refusing update from remote peer 21f0cdf4ac7e43138e291e2e0a70bc93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:08.612097 2693 consensus_queue.cc:1035] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "589e62e73d3e42d4ac54b4d884a29c69" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:08.612181 2688 consensus_queue.cc:1035] T 12655b7318714f3a9abd69999031f6aa P 21f0cdf4ac7e43138e291e2e0a70bc93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cc1738d4efa41c4b6344d164c27a90e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:00:08.616123 2686 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 19:00:10.132220 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:10.132319 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:10.133158 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7efcb76217354658b2a7044a0e0b9a1b: Sending DeleteTablet for 3 replicas of tablet 12655b7318714f3a9abd69999031f6aa
I20260925 19:00:10.133402 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 12655b7318714f3a9abd69999031f6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45140
I20260925 19:00:10.133447 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 12655b7318714f3a9abd69999031f6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52948
I20260925 19:00:10.133527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 2194
I20260925 19:00:10.133567 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 12655b7318714f3a9abd69999031f6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60418
I20260925 19:00:10.134204 2761 tablet_replica.cc:330] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69: stopping tablet replica
I20260925 19:00:10.134297 2761 raft_consensus.cc:2229] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:10.134338 2761 raft_consensus.cc:2258] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:10.134558 2761 ts_tablet_manager.cc:1905] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:10.134600 2762 tablet_replica.cc:330] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e: stopping tablet replica
I20260925 19:00:10.134676 2762 raft_consensus.cc:2229] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:10.134734 2762 raft_consensus.cc:2258] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:10.134950 2762 ts_tablet_manager.cc:1905] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:10.136399 2761 ts_tablet_manager.cc:1918] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:00:10.136441 2762 ts_tablet_manager.cc:1918] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 19:00:10.136493 2761 log.cc:1196] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/12655b7318714f3a9abd69999031f6aa
I20260925 19:00:10.136538 2762 log.cc:1196] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/12655b7318714f3a9abd69999031f6aa
I20260925 19:00:10.136719 2761 ts_tablet_manager.cc:1939] T 12655b7318714f3a9abd69999031f6aa P 589e62e73d3e42d4ac54b4d884a29c69: Deleting consensus metadata
I20260925 19:00:10.136790 2762 ts_tablet_manager.cc:1939] T 12655b7318714f3a9abd69999031f6aa P 9cc1738d4efa41c4b6344d164c27a90e: Deleting consensus metadata
I20260925 19:00:10.137060 2106 catalog_manager.cc:4841] TS 589e62e73d3e42d4ac54b4d884a29c69 (127.0.98.2:42591): tablet 12655b7318714f3a9abd69999031f6aa (table test-table [id=3962770ad7b64ca2a115cb842edcf844]) successfully deleted
I20260925 19:00:10.137102 2109 catalog_manager.cc:4841] TS 9cc1738d4efa41c4b6344d164c27a90e (127.0.98.3:35667): tablet 12655b7318714f3a9abd69999031f6aa (table test-table [id=3962770ad7b64ca2a115cb842edcf844]) successfully deleted
W20260925 19:00:10.140692 2106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40503 (error 108)
I20260925 19:00:10.140938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 2334
I20260925 19:00:10.141659 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:10.141027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40503 (local address 127.0.0.1:32796)
0925 19:00:10.141566 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:10.141613 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":62,"threads_started":1}
W20260925 19:00:10.141919 2106 catalog_manager.cc:4584] TS 21f0cdf4ac7e43138e291e2e0a70bc93 (127.0.98.1:40503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12655b7318714f3a9abd69999031f6aa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40503: connect: Connection refused (error 111)
I20260925 19:00:10.146971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 2474
I20260925 19:00:10.152112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 2090
2026-09-25T19:00:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5831 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:10 dist-test-slave-49bz krb5kdc[2767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:10 dist-test-slave-49bz krb5kdc[2767](info): set up 2 sockets
Sep 25 19:00:10 dist-test-slave-49bz krb5kdc[2767](info): commencing operation
krb5kdc: starting...
W20260925 19:00:11.153776 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:11 dist-test-slave-49bz krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:00:11.260505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43359
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:11.339351 2780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:11.339542 2780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:11.339564 2780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:11.340610 2780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:11.340646 2780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:11.340658 2780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:11.340670 2780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:11.340682 2780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:11.342024 2780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:11.342192 2780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:11.342414 2780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:11.347172 2785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:11.347196 2786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:11.347306 2780 server_base.cc:1034] running on GCE node
W20260925 19:00:11.347445 2788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:11.347700 2780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:11.348009 2780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:11.349175 2780 hybrid_clock.cc:648] HybridClock initialized: now 1790362811349156 us; error 35 us; skew 500 ppm
Sep 25 19:00:11 dist-test-slave-49bz krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:11.351554 2780 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:00:11.352031 2780 webserver.cc:460] Webserver started at http://127.0.98.62:42587/ using document root <none> and password file <none>
I20260925 19:00:11.352258 2780 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:11.352303 2780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:11.352421 2780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:11.353437 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "e2728af257c144108a88466a5076106c"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "ee69867101fb7924ed13a3d2d1514b5b"
server_key_iv: "e43d9df0e18afda96dee5b776c877652"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.353801 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2728af257c144108a88466a5076106c"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "ee69867101fb7924ed13a3d2d1514b5b"
server_key_iv: "e43d9df0e18afda96dee5b776c877652"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.355299 2780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:11.356001 2795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.356195 2780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.356266 2780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "e2728af257c144108a88466a5076106c"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "ee69867101fb7924ed13a3d2d1514b5b"
server_key_iv: "e43d9df0e18afda96dee5b776c877652"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.356361 2780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:11.375844 2780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:11.378154 2780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:11.378331 2780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:11.381927 2780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43359
I20260925 19:00:11.381978 2847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43359 every 8 connection(s)
I20260925 19:00:11.382514 2780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:11.383174 2848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:11.384550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 2780
I20260925 19:00:11.384640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:11.384810 392 external_mini_cluster.cc:1428] Setting key c443ac5b2bd1530ec73989f8fb7b6171
I20260925 19:00:11.385675 2848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c: Bootstrap starting.
I20260925 19:00:11.386337 2848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:11.386664 2848 log.cc:824] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:11.388312 2848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c: No bootstrap required, opened a new log
I20260925 19:00:11.390883 2848 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2728af257c144108a88466a5076106c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } }
I20260925 19:00:11.391029 2848 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:11.391052 2848 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2728af257c144108a88466a5076106c, State: Initialized, Role: FOLLOWER
I20260925 19:00:11.391194 2848 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2728af257c144108a88466a5076106c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } }
I20260925 19:00:11.391252 2848 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:11.391273 2848 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:11.391336 2848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [term 0 FOLLOWER]: Advancing to term 1
Sep 25 19:00:11 dist-test-slave-49bz krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:11.391995 2848 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2728af257c144108a88466a5076106c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } }
I20260925 19:00:11.392125 2848 leader_election.cc:302] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2728af257c144108a88466a5076106c; no voters:
I20260925 19:00:11.392279 2848 leader_election.cc:288] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:11.392450 2854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:11.392544 2848 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:11.392601 2854 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [term 1 LEADER]: Becoming Leader. State: Replica: e2728af257c144108a88466a5076106c, State: Running, Role: LEADER
I20260925 19:00:11.392676 2854 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2728af257c144108a88466a5076106c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } }
I20260925 19:00:11.392987 2854 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2728af257c144108a88466a5076106c. Latest consensus state: current_term: 1 leader_uuid: "e2728af257c144108a88466a5076106c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2728af257c144108a88466a5076106c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } }
I20260925 19:00:11.393085 2854 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:11.393271 2862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:11.393764 2862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:11.393869 2853 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2728af257c144108a88466a5076106c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2728af257c144108a88466a5076106c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } }
I20260925 19:00:11.393950 2853 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:11.395501 2862 catalog_manager.cc:1347] Generated new cluster ID: a5649af5277546d382b6fd4df37d5842
I20260925 19:00:11.395556 2862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:11.395669 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:11.385329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.0.98.62:43359)
0925 19:00:11.385480 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:11.385488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:11.385520 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:00:11.385946 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:11.385953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:11.385997 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:11.386108 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:11.386915 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.387739 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.388732 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.389139 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.389219 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:11.391977 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:11.391981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:11.391982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:11.391994 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:11.393768 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:11.394225 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:11.394228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:11.394228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:11.394356 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:11.394620 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:11.394621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:11.394621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:11.394908 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:11.395057 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:11.395338 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 19:00:11.395473 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260925 19:00:11.449940 2862 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:11.450706 2862 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:11.468919 2862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2728af257c144108a88466a5076106c: Generated new TSK 0
I20260925 19:00:11.469118 2862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:00:11.505812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43359
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:11.584218 2874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:11.584427 2874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:11.584462 2874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:11.585528 2874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:11.585589 2874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:11.585630 2874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:11.587045 2874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:11.587298 2874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:11.587505 2874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:00:11.592226 2874 server_base.cc:1034] running on GCE node
W20260925 19:00:11.592209 2882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:11.592202 2879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:11.592206 2880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:11.592731 2874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:11.593027 2874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:11.594203 2874 hybrid_clock.cc:648] HybridClock initialized: now 1790362811594195 us; error 38 us; skew 500 ppm
Sep 25 19:00:11 dist-test-slave-49bz krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:11.596460 2874 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:00:11.596848 2874 webserver.cc:460] Webserver started at http://127.0.98.1:34069/ using document root <none> and password file <none>
I20260925 19:00:11.597054 2874 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:11.597117 2874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:11.597256 2874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:11.598177 2874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "5f95c31f5bc2450d9f23847965e494f9"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "b17ecc75828a8da5d2bd65d7f040380a"
server_key_iv: "bf645ffe72f9a300ecd23d26e0a375a7"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.598600 2874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5f95c31f5bc2450d9f23847965e494f9"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "b17ecc75828a8da5d2bd65d7f040380a"
server_key_iv: "bf645ffe72f9a300ecd23d26e0a375a7"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.599918 2874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:11.600683 2889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.600894 2874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:11.600991 2874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "5f95c31f5bc2450d9f23847965e494f9"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "b17ecc75828a8da5d2bd65d7f040380a"
server_key_iv: "bf645ffe72f9a300ecd23d26e0a375a7"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.601065 2874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:11.618453 2874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:11.620147 2874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:11.620306 2874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:11.620502 2874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:11.620777 2874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:11.620828 2874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.620865 2874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:11.620893 2874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.626066 2874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34783
I20260925 19:00:11.626158 3002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34783 every 8 connection(s)
I20260925 19:00:11.626441 2874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:00:11.630292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 2874
I20260925 19:00:11.630383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:11.630566 392 external_mini_cluster.cc:1428] Setting key 9b54e65fa8a0a78ff8974ffdda6a1220
Sep 25 19:00:11 dist-test-slave-49bz krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:11.636195 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:11.627084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51735 (local address 127.0.98.62:43359)
0925 19:00:11.627146 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:11.627148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:11.627349 (+ 201us) server_negotiation.cc:407] Connection header received
0925 19:00:11.628222 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:11.628223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:11.628234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:11.628284 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:11.628890 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.629394 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.630591 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.630907 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.630958 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:11.633160 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:11.633161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:11.633161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:11.633169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:11.634703 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:11.635151 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:11.635152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:11.635153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:11.635213 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:11.635481 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:11.635482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:11.635483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:11.635627 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:11.635703 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:11.635971 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 19:00:11.636058 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:00:11.636266 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:11.627061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43359 (local address 127.0.98.1:51735)
0925 19:00:11.627232 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:11.627284 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:11.628121 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:11.628357 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:11.628360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:11.628617 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:11.628782 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:11.628786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.629485 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.629486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:11.630501 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:11.630504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.630982 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.630983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:11.631133 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:11.631574 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:11.631583 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:11.633069 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:11.634790 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:11.634798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:11.634800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:11.635050 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:11.635295 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:11.635296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:11.635296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:11.635413 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:11.635698 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:11.635702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:11.635834 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 19:00:11.635931 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 19:00:11.636064 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260925 19:00:11.636811 3003 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43359
I20260925 19:00:11.636907 3003 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:11.637081 3003 heartbeater.cc:511] Master 127.0.98.62:43359 requested a full tablet report, sending...
I20260925 19:00:11.637517 2812 ts_manager.cc:194] Registered new tserver with Master: 5f95c31f5bc2450d9f23847965e494f9 (127.0.98.1:34783)
I20260925 19:00:11.638109 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51735
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:00:11.654445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43359
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:11.730365 3008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:11.730558 3008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:11.730580 3008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:11.731702 3008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:11.731747 3008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:11.731798 3008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:11.733250 3008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:11.733410 3008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:11.733629 3008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:11.738404 3014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:11.738457 3016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:11.738584 3008 server_base.cc:1034] running on GCE node
W20260925 19:00:11.738404 3013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:11.738873 3008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:11.739113 3008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:11.740250 3008 hybrid_clock.cc:648] HybridClock initialized: now 1790362811740239 us; error 29 us; skew 500 ppm
Sep 25 19:00:11 dist-test-slave-49bz krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:11.742499 3008 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:00:11.742898 3008 webserver.cc:460] Webserver started at http://127.0.98.2:39763/ using document root <none> and password file <none>
I20260925 19:00:11.743098 3008 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:11.743141 3008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:11.743275 3008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:11.744241 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "82f963657c3f46ec9ce6dd4af43febbb"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "68b036fbb3efaab3bd68cc2b2663e5f0"
server_key_iv: "bd5941b951f0de9f7dd096f0f29f8137"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.744581 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "82f963657c3f46ec9ce6dd4af43febbb"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "68b036fbb3efaab3bd68cc2b2663e5f0"
server_key_iv: "bd5941b951f0de9f7dd096f0f29f8137"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.745805 3008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:00:11.746565 3023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.746773 3008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:11.746856 3008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "82f963657c3f46ec9ce6dd4af43febbb"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "68b036fbb3efaab3bd68cc2b2663e5f0"
server_key_iv: "bd5941b951f0de9f7dd096f0f29f8137"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.746924 3008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:11.773530 3008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:11.775230 3008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:11.775395 3008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:11.775604 3008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:11.775887 3008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:11.775916 3008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.775935 3008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:11.775981 3008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.781579 3008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44349
I20260925 19:00:11.781625 3136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44349 every 8 connection(s)
I20260925 19:00:11.781927 3008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:00:11 dist-test-slave-49bz krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:11.788537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 3008
I20260925 19:00:11.788637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:00:11.788851 392 external_mini_cluster.cc:1428] Setting key 429a1cd199c580999742e6010c49cfda
I20260925 19:00:11.791457 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:11.782645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36257 (local address 127.0.98.62:43359)
0925 19:00:11.782857 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:11.782860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:11.783103 (+ 243us) server_negotiation.cc:407] Connection header received
0925 19:00:11.783969 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:11.783970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:11.783980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:11.784036 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:11.784602 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.785061 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.785954 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.786119 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.786165 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:11.788483 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:11.788485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:11.788485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:11.788491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:11.789992 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:11.790479 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:11.790481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:11.790481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:11.790550 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:11.790817 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:11.790818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:11.790818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:11.790924 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:11.791006 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:11.791294 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 19:00:11.791368 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260925 19:00:11.791618 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:11.782566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43359 (local address 127.0.98.2:36257)
0925 19:00:11.782959 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:11.782999 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:11.783882 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:11.784109 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:11.784113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:11.784361 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:11.784528 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:11.784531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.785132 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.785133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:11.785883 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:11.785885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.786201 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.786202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:11.786319 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:11.786828 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:11.786836 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:11.788384 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:11.790096 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:11.790103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:11.790105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:11.790393 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:11.790617 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:11.790618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:11.790619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:11.790734 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:11.791012 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:11.791040 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:11.791161 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 19:00:11.791259 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 19:00:11.791402 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":323,"threads_started":1}
I20260925 19:00:11.792073 3137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43359
I20260925 19:00:11.792184 3137 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:11.792356 3137 heartbeater.cc:511] Master 127.0.98.62:43359 requested a full tablet report, sending...
I20260925 19:00:11.792688 2812 ts_manager.cc:194] Registered new tserver with Master: 82f963657c3f46ec9ce6dd4af43febbb (127.0.98.2:44349)
I20260925 19:00:11.793306 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36257
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:00:11.811283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43359
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:11.886745 3142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:11.886935 3142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:11.886969 3142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:11.887974 3142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:11.888031 3142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:11.888077 3142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:11.889519 3142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:11.889783 3142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:11.889974 3142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:11.894435 3150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:11.894510 3142 server_base.cc:1034] running on GCE node
W20260925 19:00:11.894395 3148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:11.894606 3147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:11.894814 3142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:11.895030 3142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:11.896178 3142 hybrid_clock.cc:648] HybridClock initialized: now 1790362811896163 us; error 28 us; skew 500 ppm
Sep 25 19:00:11 dist-test-slave-49bz krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:11.898551 3142 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:00:11.899000 3142 webserver.cc:460] Webserver started at http://127.0.98.3:43273/ using document root <none> and password file <none>
I20260925 19:00:11.899233 3142 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:11.899281 3142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:11.899399 3142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:11.900372 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ac8815af015449f83f7d8d749d9214c"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "ac79b4dba48f78328811b42f4d757cfb"
server_key_iv: "2598bc9b4b5eb4749ed201ac4af6b7b2"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.900700 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ac8815af015449f83f7d8d749d9214c"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "ac79b4dba48f78328811b42f4d757cfb"
server_key_iv: "2598bc9b4b5eb4749ed201ac4af6b7b2"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.901903 3142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:00:11.902616 3157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.902774 3142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.902863 3142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "6ac8815af015449f83f7d8d749d9214c"
format_stamp: "Formatted at 2026-09-25 19:00:11 on dist-test-slave-49bz"
server_key: "ac79b4dba48f78328811b42f4d757cfb"
server_key_iv: "2598bc9b4b5eb4749ed201ac4af6b7b2"
server_key_version: "encryptionkey@0"
I20260925 19:00:11.902930 3142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:11.949136 3142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:11.951398 3142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:11.951537 3142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:11.951747 3142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:11.952024 3142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:11.952077 3142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.952119 3142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:11.952145 3142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:11.957547 3142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38561
I20260925 19:00:11.957600 3270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38561 every 8 connection(s)
I20260925 19:00:11.957896 3142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:00:11 dist-test-slave-49bz krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:11.965665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 3142
I20260925 19:00:11.965758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:00:11.965938 392 external_mini_cluster.cc:1428] Setting key 86539ef18ea55218a23b9e05675f56d1
I20260925 19:00:11.967025 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:11.958647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36347 (local address 127.0.98.62:43359)
0925 19:00:11.958700 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:11.958702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:11.959425 (+ 723us) server_negotiation.cc:407] Connection header received
0925 19:00:11.960412 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:11.960414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:11.960426 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:11.960469 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:11.961025 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.961525 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.962470 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.962674 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.962753 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:11.964479 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:11.964481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:11.964481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:11.964488 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:11.965549 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:11.965992 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:11.965993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:11.965993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:11.966091 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:11.966425 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:11.966426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:11.966426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:11.966567 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:11.966625 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:11.966909 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 19:00:11.966968 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:00:11.967243 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:11.958616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43359 (local address 127.0.98.3:36347)
0925 19:00:11.959289 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:11.959323 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:11.960311 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:11.960541 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:11.960545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:11.960766 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:11.960935 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:11.960939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.961614 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.961616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:11.962386 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:11.962388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:11.962760 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:11.962762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:11.962871 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:11.963222 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:11.963229 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:11.964393 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:11.965628 (+ 1235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:11.965634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:11.965636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:11.965910 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:11.966165 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:11.966166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:11.966167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:11.966350 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:11.966627 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:11.966634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:11.966762 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 19:00:11.966882 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 19:00:11.967005 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":76,"threads_started":1}
I20260925 19:00:11.967618 3271 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43359
I20260925 19:00:11.967686 3271 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:11.967880 3271 heartbeater.cc:511] Master 127.0.98.62:43359 requested a full tablet report, sending...
I20260925 19:00:11.968252 2812 ts_manager.cc:194] Registered new tserver with Master: 6ac8815af015449f83f7d8d749d9214c (127.0.98.3:38561)
I20260925 19:00:11.968623 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36347
I20260925 19:00:11.977653 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 19:00:11.987922 2851 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:00:11.987170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.0.98.62:43359)
0925 19:00:11.987250 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:11.987252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:11.987304 (+ 52us) server_negotiation.cc:407] Connection header received
0925 19:00:11.987341 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:11.987342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:11.987350 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:11.987395 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:11.987866 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55390: BlockingRecv error: recv got EOF from 127.0.0.1:55390 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 19:00:11.988725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 2874
I20260925 19:00:11.993180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 3008
I20260925 19:00:11.997068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 3142
I20260925 19:00:12.001055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 2780
2026-09-25T19:00:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1847 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:12 dist-test-slave-49bz krb5kdc[3283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:12 dist-test-slave-49bz krb5kdc[3283](info): set up 2 sockets
Sep 25 19:00:12 dist-test-slave-49bz krb5kdc[3283](info): commencing operation
krb5kdc: starting...
W20260925 19:00:13.006008 392 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:13 dist-test-slave-49bz krb5kdc[3283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:00:13.113399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34829
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:13.189425 3296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:13.189651 3296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:13.189687 3296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:13.190909 3296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:13.190968 3296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:13.190992 3296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:13.191020 3296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:13.191043 3296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:13.192459 3296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:13.192713 3296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:13.192915 3296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:13.198601 3304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:13.198653 3302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:13.198570 3301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:13.198570 3296 server_base.cc:1034] running on GCE node
I20260925 19:00:13.199011 3296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:13.199295 3296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:13.200453 3296 hybrid_clock.cc:648] HybridClock initialized: now 1790362813200439 us; error 48 us; skew 500 ppm
Sep 25 19:00:13 dist-test-slave-49bz krb5kdc[3283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:13.202708 3296 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:00:13.203164 3296 webserver.cc:460] Webserver started at http://127.0.98.62:43879/ using document root <none> and password file <none>
I20260925 19:00:13.203373 3296 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:13.203419 3296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:13.203564 3296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:13.204537 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "267654dac56a459b9a53370030ef2896"
format_stamp: "Formatted at 2026-09-25 19:00:13 on dist-test-slave-49bz"
server_key: "3040d631cf7a01a10a2f764c8317023c"
server_key_iv: "84f7ecda9b02fc19d0437c13e03dea57"
server_key_version: "encryptionkey@0"
I20260925 19:00:13.204871 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "267654dac56a459b9a53370030ef2896"
format_stamp: "Formatted at 2026-09-25 19:00:13 on dist-test-slave-49bz"
server_key: "3040d631cf7a01a10a2f764c8317023c"
server_key_iv: "84f7ecda9b02fc19d0437c13e03dea57"
server_key_version: "encryptionkey@0"
I20260925 19:00:13.206077 3296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 19:00:13.206867 3311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:13.207055 3296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:00:13.207145 3296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "267654dac56a459b9a53370030ef2896"
format_stamp: "Formatted at 2026-09-25 19:00:13 on dist-test-slave-49bz"
server_key: "3040d631cf7a01a10a2f764c8317023c"
server_key_iv: "84f7ecda9b02fc19d0437c13e03dea57"
server_key_version: "encryptionkey@0"
I20260925 19:00:13.207211 3296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:13.225417 3296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:13.227180 3296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:13.227313 3296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:13.231207 3296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34829
I20260925 19:00:13.231274 3363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34829 every 8 connection(s)
I20260925 19:00:13.231554 3296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:13.232352 3364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:13.234561 3364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896: Bootstrap starting.
I20260925 19:00:13.235100 3364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:13.235388 3364 log.cc:824] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:13.235989 3364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896: No bootstrap required, opened a new log
I20260925 19:00:13.237226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 3296
I20260925 19:00:13.237321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:13.237479 392 external_mini_cluster.cc:1428] Setting key 1a6afc1be5502b8b20055c66a93d2816
I20260925 19:00:13.237896 3364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267654dac56a459b9a53370030ef2896" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260925 19:00:13.238106 3364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:13.238166 3364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267654dac56a459b9a53370030ef2896, State: Initialized, Role: FOLLOWER
I20260925 19:00:13.238308 3364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267654dac56a459b9a53370030ef2896" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260925 19:00:13.238377 3364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:13.238399 3364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:13.238464 3364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:13.239089 3364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267654dac56a459b9a53370030ef2896" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260925 19:00:13.239202 3364 leader_election.cc:302] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 267654dac56a459b9a53370030ef2896; no voters:
I20260925 19:00:13.239362 3364 leader_election.cc:288] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:13.239421 3369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:13.239585 3369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [term 1 LEADER]: Becoming Leader. State: Replica: 267654dac56a459b9a53370030ef2896, State: Running, Role: LEADER
I20260925 19:00:13.239612 3364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:13.239679 3369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267654dac56a459b9a53370030ef2896" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260925 19:00:13.239962 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 267654dac56a459b9a53370030ef2896. Latest consensus state: current_term: 1 leader_uuid: "267654dac56a459b9a53370030ef2896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267654dac56a459b9a53370030ef2896" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } } }
I20260925 19:00:13.239972 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "267654dac56a459b9a53370030ef2896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267654dac56a459b9a53370030ef2896" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } } }
I20260925 19:00:13.240134 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:13.240141 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:13.240374 3378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:13.240890 3378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:13.243227 3378 catalog_manager.cc:1347] Generated new cluster ID: 5cb38e8f4d0b4c5cbcb7c1c195976ca7
I20260925 19:00:13.243285 3378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:00:13 dist-test-slave-49bz krb5kdc[3283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:13.247579 3378 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:13.248251 3378 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:13.249099 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:13.237936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.0.98.62:34829)
0925 19:00:13.238522 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:13.238531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:13.238565 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:00:13.238663 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:13.238667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:13.238734 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:13.238836 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:13.239688 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:13.241149 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:13.241962 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:13.242600 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:13.242713 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:13.245496 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:13.245499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:13.245500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:13.245511 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:13.247405 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:13.247820 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:13.247823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:13.247823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:13.247957 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:13.248189 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:13.248190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:13.248190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:13.248487 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:13.248657 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:13.248762 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 19:00:13.248909 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":496,"threads_started":1}
I20260925 19:00:13.250025 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:00:13.255167 3378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 267654dac56a459b9a53370030ef2896: Generated new TSK 0
I20260925 19:00:13.255311 3378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:00:13.263916 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:13.260455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.0.98.62:34829)
0925 19:00:13.260508 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:13.260510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:13.260597 (+ 87us) server_negotiation.cc:407] Connection header received
0925 19:00:13.260625 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:13.260626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:13.260638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:13.260707 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:13.260987 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:13.261740 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:13.262491 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:13.262831 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:13.262895 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:13.263489 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:13.263490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:13.263490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:13.263694 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:13.263710 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:13.263716 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:13.263776 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:13.263822 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 19:00:13.263846 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:00:13.265887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 3296
2026-09-25T19:00:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1262 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:13 dist-test-slave-49bz krb5kdc[3397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:13 dist-test-slave-49bz krb5kdc[3397](info): set up 2 sockets
Sep 25 19:00:13 dist-test-slave-49bz krb5kdc[3397](info): commencing operation
krb5kdc: starting...
W20260925 19:00:14.265877 392 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:14 dist-test-slave-49bz krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:00:14.372370 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:14.449307 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:14.449486 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:14.449507 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:14.450640 3410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:14.450677 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:14.450691 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:14.450703 3410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:14.450716 3410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:14.452127 3410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:14.452370 3410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:14.452593 3410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:14.457275 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:14.457279 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:14.457279 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:14.457585 3410 server_base.cc:1034] running on GCE node
I20260925 19:00:14.457821 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:14.458076 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:14.459237 3410 hybrid_clock.cc:648] HybridClock initialized: now 1790362814459211 us; error 53 us; skew 500 ppm
Sep 25 19:00:14 dist-test-slave-49bz krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:14.461709 3410 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:00:14.462128 3410 webserver.cc:460] Webserver started at http://127.0.98.62:40197/ using document root <none> and password file <none>
I20260925 19:00:14.462390 3410 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:14.462453 3410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:14.462585 3410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:14.463514 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "282085a493404df58a470f7f0bb72e1d"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "9306370d8ce717b90f5763a9079d5b81"
server_key_iv: "6ae0c42d6467e1cdae6af4e2c6852b4f"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.463837 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "282085a493404df58a470f7f0bb72e1d"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "9306370d8ce717b90f5763a9079d5b81"
server_key_iv: "6ae0c42d6467e1cdae6af4e2c6852b4f"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.465070 3410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:00:14.465826 3425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:14.465965 3410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:14.466041 3410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "282085a493404df58a470f7f0bb72e1d"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "9306370d8ce717b90f5763a9079d5b81"
server_key_iv: "6ae0c42d6467e1cdae6af4e2c6852b4f"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.466089 3410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:14.477584 3410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:14.479655 3410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:14.479756 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:14.483592 3410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38417
I20260925 19:00:14.483664 3477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38417 every 8 connection(s)
I20260925 19:00:14.483958 3410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:14.484772 3478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:14.486047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 3410
I20260925 19:00:14.486126 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:14.486336 392 external_mini_cluster.cc:1428] Setting key b92c1d27a6cd3d93257d49832db771ab
I20260925 19:00:14.487129 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d: Bootstrap starting.
I20260925 19:00:14.487676 3478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:14.487943 3478 log.cc:824] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:14.488724 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d: No bootstrap required, opened a new log
I20260925 19:00:14.491292 3478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282085a493404df58a470f7f0bb72e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260925 19:00:14.491433 3478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:14.491464 3478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282085a493404df58a470f7f0bb72e1d, State: Initialized, Role: FOLLOWER
I20260925 19:00:14.491593 3478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282085a493404df58a470f7f0bb72e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260925 19:00:14.491711 3478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:14.491742 3478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:14.491783 3478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:14.492280 3478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282085a493404df58a470f7f0bb72e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260925 19:00:14.492421 3478 leader_election.cc:302] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 282085a493404df58a470f7f0bb72e1d; no voters:
I20260925 19:00:14.492571 3478 leader_election.cc:288] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:14.492620 3483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:14.492779 3483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [term 1 LEADER]: Becoming Leader. State: Replica: 282085a493404df58a470f7f0bb72e1d, State: Running, Role: LEADER
I20260925 19:00:14.492817 3478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:14.492926 3483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282085a493404df58a470f7f0bb72e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
Sep 25 19:00:14 dist-test-slave-49bz krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:14.494067 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "282085a493404df58a470f7f0bb72e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282085a493404df58a470f7f0bb72e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260925 19:00:14.494203 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:14.494370 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282085a493404df58a470f7f0bb72e1d. Latest consensus state: current_term: 1 leader_uuid: "282085a493404df58a470f7f0bb72e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282085a493404df58a470f7f0bb72e1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260925 19:00:14.494441 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d [sys.catalog]: This master's current role is: LEADER
W20260925 19:00:14.495013 3499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:00:14.495092 3499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:00:14.495175 3498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:14.495363 3498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:14.497059 3498 catalog_manager.cc:1347] Generated new cluster ID: 56f874313f2a44b5ae81fd33fba770b0
I20260925 19:00:14.497148 3498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:14.497854 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:14.486855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.0.98.62:38417)
0925 19:00:14.487017 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:14.487024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:14.487044 (+ 20us) server_negotiation.cc:407] Connection header received
0925 19:00:14.487479 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:14.487486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:14.487541 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:14.487690 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:14.488411 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.489799 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.490887 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.491504 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.491623 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:14.494376 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:14.494380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:14.494380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:14.494392 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:14.496306 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:14.496712 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:14.496714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:14.496714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:14.496786 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:14.497001 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:14.497002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:14.497003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:14.497292 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:14.497448 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:14.497567 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 19:00:14.497696 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260925 19:00:14.513741 3498 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:14.514389 3498 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:14.521863 3498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 282085a493404df58a470f7f0bb72e1d: Generated new TSK 0
I20260925 19:00:14.522013 3498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:00:14.554224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:14.631311 3504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:14.631523 3504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:14.631559 3504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:14.632644 3504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:14.632705 3504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:14.632746 3504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:14.634121 3504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:14.634403 3504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:14.634616 3504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:14.639289 3512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:14.639302 3510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:14.639317 3509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:14.639433 3504 server_base.cc:1034] running on GCE node
I20260925 19:00:14.639776 3504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:14.640013 3504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:14.641187 3504 hybrid_clock.cc:648] HybridClock initialized: now 1790362814641181 us; error 30 us; skew 500 ppm
Sep 25 19:00:14 dist-test-slave-49bz krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:14.643633 3504 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:00:14.644115 3504 webserver.cc:460] Webserver started at http://127.0.98.1:36119/ using document root <none> and password file <none>
I20260925 19:00:14.644330 3504 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:14.644372 3504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:14.644517 3504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:14.645529 3504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "4687cba9cdf1483b9e7b5010cdac62e1"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "4378a91673cd02cd0f729104e1447cb8"
server_key_iv: "4b56cbda620f2c0e89834dc4d87b6b3b"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.645887 3504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4687cba9cdf1483b9e7b5010cdac62e1"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "4378a91673cd02cd0f729104e1447cb8"
server_key_iv: "4b56cbda620f2c0e89834dc4d87b6b3b"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.647168 3504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:14.647820 3519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:14.647974 3504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:14.648048 3504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "4687cba9cdf1483b9e7b5010cdac62e1"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "4378a91673cd02cd0f729104e1447cb8"
server_key_iv: "4b56cbda620f2c0e89834dc4d87b6b3b"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.648096 3504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:14.679515 3504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:14.681818 3504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:14.681958 3504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:14.682191 3504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:14.682502 3504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:14.682564 3504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:14.682601 3504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:14.682626 3504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:14.688074 3504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46185
I20260925 19:00:14.688113 3632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46185 every 8 connection(s)
I20260925 19:00:14.688387 3504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:00:14 dist-test-slave-49bz krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:14.696590 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:14.689051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55407 (local address 127.0.98.62:38417)
0925 19:00:14.689096 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:14.689098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:14.689277 (+ 179us) server_negotiation.cc:407] Connection header received
0925 19:00:14.689736 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:14.689737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:14.689746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:14.689799 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:14.690787 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.691277 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.692182 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.692372 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.692439 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:14.694411 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:14.694412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:14.694412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:14.694418 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:14.695509 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:14.695857 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:14.695857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:14.695858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:14.695912 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:14.696119 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:14.696119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:14.696120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:14.696220 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:14.696267 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:14.696471 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 19:00:14.696531 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:00:14.696771 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:14.689023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:55407)
0925 19:00:14.689168 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:14.689199 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:14.689642 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:14.689850 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:14.689854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:14.690502 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:14.690711 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:14.690716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.691352 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.691352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:14.692112 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:14.692114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.692455 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.692456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:14.692545 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:14.693008 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:14.693015 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:14.694338 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:14.695571 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:14.695575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:14.695577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:14.695796 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:14.695982 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:14.695982 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:14.695983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:14.696069 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:14.696271 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:14.696274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:14.696374 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 19:00:14.696448 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:00:14.696550 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260925 19:00:14.697273 3633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260925 19:00:14.697342 3633 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:14.697458 3633 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260925 19:00:14.697882 3442 ts_manager.cc:194] Registered new tserver with Master: 4687cba9cdf1483b9e7b5010cdac62e1 (127.0.98.1:46185)
I20260925 19:00:14.698508 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55407
I20260925 19:00:14.698603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 3504
I20260925 19:00:14.698691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:14.698907 392 external_mini_cluster.cc:1428] Setting key 6952833c59e728e72558bb2ecb6e5692
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:00:14.720487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:14.795261 3638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:14.795447 3638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:14.795499 3638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:14.796574 3638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:14.796609 3638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:14.796635 3638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:14.797912 3638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:14.798081 3638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:14.798235 3638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:14.802680 3646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:14.802685 3643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:14.802711 3644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:14.802858 3638 server_base.cc:1034] running on GCE node
I20260925 19:00:14.803143 3638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:14.803376 3638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:14.804528 3638 hybrid_clock.cc:648] HybridClock initialized: now 1790362814804519 us; error 38 us; skew 500 ppm
Sep 25 19:00:14 dist-test-slave-49bz krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:14.806830 3638 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:00:14.807217 3638 webserver.cc:460] Webserver started at http://127.0.98.2:43879/ using document root <none> and password file <none>
I20260925 19:00:14.807420 3638 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:14.807482 3638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:14.807622 3638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:14.808565 3638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "689f35c6d9cc4d19a6ba5ed06280c59b"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "8904d140e7bcdb890a432b63e378f8bb"
server_key_iv: "52d13da2526971b55868e6a924b769eb"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.808925 3638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "689f35c6d9cc4d19a6ba5ed06280c59b"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "8904d140e7bcdb890a432b63e378f8bb"
server_key_iv: "52d13da2526971b55868e6a924b769eb"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.810171 3638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:00:14.810931 3653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:14.811100 3638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:00:14.811194 3638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "689f35c6d9cc4d19a6ba5ed06280c59b"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "8904d140e7bcdb890a432b63e378f8bb"
server_key_iv: "52d13da2526971b55868e6a924b769eb"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.811271 3638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:14.829285 3638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:14.831284 3638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:14.831440 3638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:14.831656 3638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:14.831938 3638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:14.831987 3638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:14.832020 3638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:14.832046 3638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:14.837570 3638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40295
I20260925 19:00:14.837610 3766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40295 every 8 connection(s)
I20260925 19:00:14.837972 3638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:00:14 dist-test-slave-49bz krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:14.844558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 3638
I20260925 19:00:14.844668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:00:14.844890 392 external_mini_cluster.cc:1428] Setting key a32efb6acd96f1a320690149c952d291
I20260925 19:00:14.847090 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:14.838695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37899 (local address 127.0.98.62:38417)
0925 19:00:14.838752 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:14.838754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:14.838977 (+ 223us) server_negotiation.cc:407] Connection header received
0925 19:00:14.839427 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:14.839427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:14.839436 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:14.839488 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:14.840331 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.840792 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.841892 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.842059 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.842107 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:14.844167 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:14.844169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:14.844169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:14.844175 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:14.845844 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:14.846201 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:14.846203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:14.846203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:14.846281 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:14.846493 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:14.846494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:14.846494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:14.846637 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:14.846702 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:14.846939 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 19:00:14.847007 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 19:00:14.847270 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:14.838664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:37899)
0925 19:00:14.838831 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:14.838885 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:14.839348 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:14.839541 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:14.839544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:14.840079 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:14.840264 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:14.840268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.840868 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.840869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:14.841815 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:14.841818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.842174 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.842175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:14.842304 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:14.842710 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:14.842718 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:14.844045 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:14.845916 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:14.845921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:14.845923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:14.846118 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:14.846334 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:14.846335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:14.846335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:14.846432 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:14.846699 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:14.846703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:14.846803 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 19:00:14.846915 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 19:00:14.847049 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260925 19:00:14.847710 3767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260925 19:00:14.847779 3767 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:14.847947 3767 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260925 19:00:14.848307 3442 ts_manager.cc:194] Registered new tserver with Master: 689f35c6d9cc4d19a6ba5ed06280c59b (127.0.98.2:40295)
I20260925 19:00:14.848677 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37899
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:00:14.869339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:14.945713 3772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:14.945935 3772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:14.945971 3772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:14.947134 3772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:14.947206 3772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:14.947249 3772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:14.948611 3772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:14.948858 3772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:14.949095 3772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:14.953567 3777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:14.953634 3780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:14.953567 3778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:14.953881 3772 server_base.cc:1034] running on GCE node
I20260925 19:00:14.954067 3772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:14.954329 3772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:14.955487 3772 hybrid_clock.cc:648] HybridClock initialized: now 1790362814955473 us; error 30 us; skew 500 ppm
Sep 25 19:00:14 dist-test-slave-49bz krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:14.957912 3772 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:00:14.958340 3772 webserver.cc:460] Webserver started at http://127.0.98.3:37033/ using document root <none> and password file <none>
I20260925 19:00:14.958552 3772 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:14.958616 3772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:14.958783 3772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:14.959792 3772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "855ac3f04e5f4b389bc897d8962bf566"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "45f966854b4269860e0167b228942aed"
server_key_iv: "b81c1c0c3a182e2c39ef5043e449b8af"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.960165 3772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "855ac3f04e5f4b389bc897d8962bf566"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "45f966854b4269860e0167b228942aed"
server_key_iv: "b81c1c0c3a182e2c39ef5043e449b8af"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.961369 3772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:14.962150 3787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:14.962345 3772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:00:14.962438 3772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "855ac3f04e5f4b389bc897d8962bf566"
format_stamp: "Formatted at 2026-09-25 19:00:14 on dist-test-slave-49bz"
server_key: "45f966854b4269860e0167b228942aed"
server_key_iv: "b81c1c0c3a182e2c39ef5043e449b8af"
server_key_version: "encryptionkey@0"
I20260925 19:00:14.962510 3772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:14.974038 3772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:14.976243 3772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:14.976389 3772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:14.976593 3772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:14.976866 3772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:14.976915 3772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:14.976953 3772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:14.976979 3772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:14.982970 3772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41259
I20260925 19:00:14.983018 3900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41259 every 8 connection(s)
I20260925 19:00:14.983307 3772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:00:14 dist-test-slave-49bz krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:14.991618 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:14.983969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52249 (local address 127.0.98.62:38417)
0925 19:00:14.984028 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:14.984030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:14.984234 (+ 204us) server_negotiation.cc:407] Connection header received
0925 19:00:14.984685 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:14.984686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:14.984695 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:14.984747 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:14.985597 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.986058 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.987097 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.987279 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.987330 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:14.989304 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:14.989305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:14.989306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:14.989312 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:14.990533 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:14.990892 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:14.990892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:14.990893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:14.990937 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:14.991139 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:14.991140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:14.991140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:14.991241 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:14.991287 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:14.991505 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 19:00:14.991559 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:00:14.991873 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:14.983935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:52249)
0925 19:00:14.984083 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:14.984146 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:14.984604 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:14.984795 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:14.984798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:14.985348 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:14.985525 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:14.985529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.986150 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.986151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:14.987027 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:14.987029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:14.987361 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:14.987362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:14.987448 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:14.987826 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:14.987834 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:14.989222 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:14.990601 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:14.990615 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:14.990616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:14.990830 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:14.990993 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:14.990994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:14.990994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:14.991085 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:14.991295 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:14.991300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:14.991403 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 19:00:14.991482 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:00:14.991590 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260925 19:00:14.992355 3901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260925 19:00:14.992450 3901 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:14.992693 3901 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260925 19:00:14.993088 3442 ts_manager.cc:194] Registered new tserver with Master: 855ac3f04e5f4b389bc897d8962bf566 (127.0.98.3:41259)
I20260925 19:00:14.993340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 3772
I20260925 19:00:14.993446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:00:14.993530 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52249
I20260925 19:00:14.993680 392 external_mini_cluster.cc:1428] Setting key 6fd34caf616843ac242b4d9802be00c7
I20260925 19:00:14.994683 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:00:15 dist-test-slave-49bz krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362815, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 19:00:15 dist-test-slave-49bz krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362815, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:15.016870 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:15.010043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.0.98.62:38417)
0925 19:00:15.010133 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:15.010135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:15.010231 (+ 96us) server_negotiation.cc:407] Connection header received
0925 19:00:15.010321 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:15.010321 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:15.010337 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:15.010393 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:15.010960 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:15.011421 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:15.012165 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:15.012333 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:15.012404 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:15.014543 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:15.014544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:15.014544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:15.014552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:15.015777 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:15.016110 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:15.016111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:15.016111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:15.016172 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:15.016362 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:15.016363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:15.016364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:15.016494 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:15.016600 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:15.016699 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 19:00:15.016775 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 19:00:15.017078 3442 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37452
I20260925 19:00:15.017617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 3504
I20260925 19:00:15.022927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 3638
I20260925 19:00:15.026975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 3772
I20260925 19:00:15.031638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 3410
2026-09-25T19:00:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1767 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:15 dist-test-slave-49bz krb5kdc[3913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:15 dist-test-slave-49bz krb5kdc[3913](info): set up 2 sockets
Sep 25 19:00:15 dist-test-slave-49bz krb5kdc[3913](info): commencing operation
krb5kdc: starting...
W20260925 19:00:16.025959 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:00:16.132799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42309
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:16.209406 3926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:16.209589 3926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:16.209611 3926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:16.210781 3926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:16.210819 3926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:16.210834 3926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:16.210845 3926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:16.210857 3926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:16.212167 3926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:16.212334 3926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:16.212484 3926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:16.216949 3932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:16.216992 3934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:16.217147 3926 server_base.cc:1034] running on GCE node
W20260925 19:00:16.217171 3931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:16.217444 3926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:16.217722 3926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:16.218902 3926 hybrid_clock.cc:648] HybridClock initialized: now 1790362816218890 us; error 25 us; skew 500 ppm
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:16.224298 3926 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:00:16.224740 3926 webserver.cc:460] Webserver started at http://127.0.98.62:34869/ using document root <none> and password file <none>
I20260925 19:00:16.224963 3926 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:16.225029 3926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:16.225173 3926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:16.226142 3926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "22e442c263a54710b597dae2872811a7"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "ef0e136415240e648bbf663888589fa5"
server_key_iv: "b03c88b0283f3daa62c5d255706e4ac6"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.226512 3926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "22e442c263a54710b597dae2872811a7"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "ef0e136415240e648bbf663888589fa5"
server_key_iv: "b03c88b0283f3daa62c5d255706e4ac6"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.227739 3926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:16.228368 3941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:16.228518 3926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:16.228605 3926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "22e442c263a54710b597dae2872811a7"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "ef0e136415240e648bbf663888589fa5"
server_key_iv: "b03c88b0283f3daa62c5d255706e4ac6"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.228689 3926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:16.245842 3926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:16.248018 3926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:16.248190 3926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:16.252089 3926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42309
I20260925 19:00:16.252117 3993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42309 every 8 connection(s)
I20260925 19:00:16.252429 3926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:16.253182 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:16.255220 3994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7: Bootstrap starting.
I20260925 19:00:16.255698 3994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:16.255947 3994 log.cc:824] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:16.256544 3994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7: No bootstrap required, opened a new log
I20260925 19:00:16.256832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 3926
I20260925 19:00:16.256938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:16.257112 392 external_mini_cluster.cc:1428] Setting key c524394e3f0e244ea1954c12a272b58f
I20260925 19:00:16.258596 3994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e442c263a54710b597dae2872811a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } }
I20260925 19:00:16.258721 3994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:16.258744 3994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22e442c263a54710b597dae2872811a7, State: Initialized, Role: FOLLOWER
I20260925 19:00:16.258862 3994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e442c263a54710b597dae2872811a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } }
I20260925 19:00:16.258941 3994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:16.258987 3994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:16.259059 3994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:16.260067 3994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e442c263a54710b597dae2872811a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } }
I20260925 19:00:16.260150 3994 leader_election.cc:302] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22e442c263a54710b597dae2872811a7; no voters:
I20260925 19:00:16.260367 3994 leader_election.cc:288] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:16.260392 3999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:16.260505 3999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [term 1 LEADER]: Becoming Leader. State: Replica: 22e442c263a54710b597dae2872811a7, State: Running, Role: LEADER
I20260925 19:00:16.260577 3999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e442c263a54710b597dae2872811a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } }
I20260925 19:00:16.260623 3994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:16.260861 3999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22e442c263a54710b597dae2872811a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e442c263a54710b597dae2872811a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } } }
I20260925 19:00:16.261022 3999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:16.261214 3999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22e442c263a54710b597dae2872811a7. Latest consensus state: current_term: 1 leader_uuid: "22e442c263a54710b597dae2872811a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e442c263a54710b597dae2872811a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } } }
I20260925 19:00:16.261335 3999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:16.261961 4006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:16.262647 4006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:16.264297 4006 catalog_manager.cc:1347] Generated new cluster ID: 01f2fb9611c04b57a2ba76a0f16647b2
I20260925 19:00:16.264349 4006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:16.269097 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:16.257567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.0.98.62:42309)
0925 19:00:16.257756 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:16.257762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:16.257785 (+ 23us) server_negotiation.cc:407] Connection header received
0925 19:00:16.258608 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:16.258615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:16.258665 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:16.258822 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:16.259725 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.260904 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.262087 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.262715 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.262821 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:16.265469 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:16.265473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:16.265473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:16.265486 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:16.267457 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.267811 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.267814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.267815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.267942 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.268143 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.268144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.268144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.268432 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:16.268583 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:16.268653 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 19:00:16.268884 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260925 19:00:16.272535 4006 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:16.272996 4006 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:16.276821 4006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22e442c263a54710b597dae2872811a7: Generated new TSK 0
I20260925 19:00:16.276975 4006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:00:16.302963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:16.379519 4020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:16.379706 4020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:16.379729 4020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:16.380841 4020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:16.380878 4020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:16.380901 4020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:16.382243 4020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:16.382519 4020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:16.382750 4020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:16.387374 4026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:16.387447 4028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:16.387544 4020 server_base.cc:1034] running on GCE node
W20260925 19:00:16.387578 4025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:16.387765 4020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:16.388017 4020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:16.389189 4020 hybrid_clock.cc:648] HybridClock initialized: now 1790362816389175 us; error 41 us; skew 500 ppm
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:16.391681 4020 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:00:16.392077 4020 webserver.cc:460] Webserver started at http://127.0.98.1:37563/ using document root <none> and password file <none>
I20260925 19:00:16.392336 4020 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:16.392402 4020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:16.392545 4020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:16.393580 4020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "515a8a8086f243ba80cb7b042c04aeda"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "fccc101f578368dcbfcfc374c509736d"
server_key_iv: "4148ef0a1a23bdad0d49b22ad3dc0ded"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.393955 4020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "515a8a8086f243ba80cb7b042c04aeda"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "fccc101f578368dcbfcfc374c509736d"
server_key_iv: "4148ef0a1a23bdad0d49b22ad3dc0ded"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.395264 4020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:16.395947 4035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:16.396111 4020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:16.396211 4020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "515a8a8086f243ba80cb7b042c04aeda"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "fccc101f578368dcbfcfc374c509736d"
server_key_iv: "4148ef0a1a23bdad0d49b22ad3dc0ded"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.396286 4020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:16.416421 4020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:16.418700 4020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:16.418855 4020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:16.419094 4020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:16.419428 4020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:16.419482 4020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:16.419516 4020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:16.419564 4020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:16.425201 4020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33947
I20260925 19:00:16.425264 4148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33947 every 8 connection(s)
I20260925 19:00:16.425599 4020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:00:16.426949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 4020
I20260925 19:00:16.427032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:16.427242 392 external_mini_cluster.cc:1428] Setting key d6e63a357da942f695e5e95eef235947
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 19:00:16.435149 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:16.426349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39905 (local address 127.0.98.62:42309)
0925 19:00:16.426401 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:16.426403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:16.426619 (+ 216us) server_negotiation.cc:407] Connection header received
0925 19:00:16.427768 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:16.427769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:16.427784 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:16.427838 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:16.428478 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.428986 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.429992 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.430213 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.430310 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:16.432492 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:16.432493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:16.432494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:16.432501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:16.433802 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.434199 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.434200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.434200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.434286 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.434517 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.434518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.434519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.434636 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:16.434713 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:16.434979 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 19:00:16.435063 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:00:16.435343 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:16.426347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42309 (local address 127.0.98.1:39905)
0925 19:00:16.426506 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:16.426539 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:16.427668 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:16.427911 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:16.427915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:16.428203 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:16.428383 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:16.428387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.429091 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.429092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:16.429891 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:16.429893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.430319 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.430320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:16.430448 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:16.430967 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:16.430974 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:16.432393 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:16.433882 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:16.433889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:16.433891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:16.434103 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:16.434344 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:16.434345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:16.434345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:16.434450 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:16.434715 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:16.434722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:16.434850 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 19:00:16.434955 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 19:00:16.435097 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260925 19:00:16.435813 4149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42309
I20260925 19:00:16.435906 4149 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:16.436074 4149 heartbeater.cc:511] Master 127.0.98.62:42309 requested a full tablet report, sending...
I20260925 19:00:16.436517 3958 ts_manager.cc:194] Registered new tserver with Master: 515a8a8086f243ba80cb7b042c04aeda (127.0.98.1:33947)
I20260925 19:00:16.437101 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:00:16.450384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:16.525751 4154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:16.525930 4154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:16.525951 4154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:16.526983 4154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:16.527016 4154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:16.527041 4154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:16.528360 4154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:16.528519 4154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:16.528669 4154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:16.533424 4162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:16.533443 4154 server_base.cc:1034] running on GCE node
W20260925 19:00:16.533416 4159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:16.533399 4160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:16.533854 4154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:16.534044 4154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:16.535166 4154 hybrid_clock.cc:648] HybridClock initialized: now 1790362816535145 us; error 37 us; skew 500 ppm
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:16.537427 4154 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:00:16.537765 4154 webserver.cc:460] Webserver started at http://127.0.98.2:44431/ using document root <none> and password file <none>
I20260925 19:00:16.537962 4154 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:16.538021 4154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:16.538187 4154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:16.539186 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "96ea2e4ed08749adbb73acdb213ac52a"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "8009f9f75d429cfe5a268a63b1ce302e"
server_key_iv: "4bc5a3e1b03c4938b384c9979a36fd20"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.539530 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "96ea2e4ed08749adbb73acdb213ac52a"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "8009f9f75d429cfe5a268a63b1ce302e"
server_key_iv: "4bc5a3e1b03c4938b384c9979a36fd20"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.540764 4154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:00:16.541441 4169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:16.541594 4154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:16.541680 4154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "96ea2e4ed08749adbb73acdb213ac52a"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "8009f9f75d429cfe5a268a63b1ce302e"
server_key_iv: "4bc5a3e1b03c4938b384c9979a36fd20"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.541755 4154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:16.549597 4154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:16.551412 4154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:16.551579 4154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:16.551789 4154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:16.552438 4154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:16.552506 4154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:16.552567 4154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:16.552595 4154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:16.560971 4154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43915
I20260925 19:00:16.561053 4282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43915 every 8 connection(s)
I20260925 19:00:16.561430 4154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:00:16.564077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 4154
I20260925 19:00:16.564174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:00:16.564364 392 external_mini_cluster.cc:1428] Setting key aa23d3dd7768b6d4700ca0499be41a04
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:16.573319 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:16.562222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45803 (local address 127.0.98.62:42309)
0925 19:00:16.562305 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:16.562307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:16.562706 (+ 399us) server_negotiation.cc:407] Connection header received
0925 19:00:16.563743 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:16.563745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:16.563759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:16.563852 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:16.564680 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.565578 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.567333 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.567651 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.567721 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:16.570558 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:16.570559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:16.570559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:16.570566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:16.571836 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.572351 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.572352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.572353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.572429 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.572684 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.572685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.572685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.572800 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:16.572875 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:16.573147 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 19:00:16.573233 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 19:00:16.573482 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:16.562431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42309 (local address 127.0.98.2:45803)
0925 19:00:16.562604 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:16.562637 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:16.563635 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:16.563940 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:16.563944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:16.564311 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:16.564582 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:16.564588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.565681 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.565682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:16.567248 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:16.567251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.567785 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.567787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:16.567926 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:16.568537 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:16.568559 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:16.570427 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:16.571929 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:16.571935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:16.571937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:16.572264 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:16.572493 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:16.572494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:16.572495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:16.572605 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:16.572865 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:16.572871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:16.573006 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 19:00:16.573101 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 19:00:16.573249 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260925 19:00:16.573915 4283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42309
I20260925 19:00:16.574054 4283 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:16.574290 4283 heartbeater.cc:511] Master 127.0.98.62:42309 requested a full tablet report, sending...
I20260925 19:00:16.574683 3958 ts_manager.cc:194] Registered new tserver with Master: 96ea2e4ed08749adbb73acdb213ac52a (127.0.98.2:43915)
I20260925 19:00:16.575202 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45803
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:00:16.598440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:16.676707 4288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:16.676873 4288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:16.676893 4288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:16.677887 4288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:16.677918 4288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:16.677942 4288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:16.679332 4288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:16.679497 4288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:16.679651 4288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:16.684294 4296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:16.684350 4294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:16.684367 4293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:16.684584 4288 server_base.cc:1034] running on GCE node
I20260925 19:00:16.684801 4288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:16.685061 4288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:16.686206 4288 hybrid_clock.cc:648] HybridClock initialized: now 1790362816686199 us; error 35 us; skew 500 ppm
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:16.688536 4288 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:00:16.688916 4288 webserver.cc:460] Webserver started at http://127.0.98.3:41565/ using document root <none> and password file <none>
I20260925 19:00:16.689141 4288 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:16.689226 4288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:16.689365 4288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:16.690411 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ea1cf12a1e74645a6e451ef23d191ec"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "04dd88e93aa3b43163dc7c674d1d358a"
server_key_iv: "30b8499531b4ad7bd20a589d800bc92b"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.690734 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ea1cf12a1e74645a6e451ef23d191ec"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "04dd88e93aa3b43163dc7c674d1d358a"
server_key_iv: "30b8499531b4ad7bd20a589d800bc92b"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.691992 4288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:00:16.692646 4303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:16.692791 4288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:16.692883 4288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "4ea1cf12a1e74645a6e451ef23d191ec"
format_stamp: "Formatted at 2026-09-25 19:00:16 on dist-test-slave-49bz"
server_key: "04dd88e93aa3b43163dc7c674d1d358a"
server_key_iv: "30b8499531b4ad7bd20a589d800bc92b"
server_key_version: "encryptionkey@0"
I20260925 19:00:16.692981 4288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:16.705718 4288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:16.707333 4288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:16.707477 4288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:16.707659 4288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:16.707927 4288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:16.707978 4288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:16.708012 4288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:16.708037 4288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:16.713443 4288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43437
I20260925 19:00:16.713470 4416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43437 every 8 connection(s)
I20260925 19:00:16.713769 4288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:16.722093 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:16.714525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54749 (local address 127.0.98.62:42309)
0925 19:00:16.714594 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:16.714596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:16.714776 (+ 180us) server_negotiation.cc:407] Connection header received
0925 19:00:16.715663 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:16.715664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:16.715677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:16.715722 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:16.716278 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.716754 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.717726 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.717909 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.718008 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:16.719997 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:16.719999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:16.719999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:16.720005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:16.721066 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.721386 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.721386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.721387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.721430 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.721641 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.721642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.721642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.721730 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:16.721776 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:16.721978 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 19:00:16.722036 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 19:00:16.722280 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:16.714473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42309 (local address 127.0.98.3:54749)
0925 19:00:16.714652 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:16.714706 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:16.715559 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:16.715781 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:16.715785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:16.716021 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:16.716179 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:16.716183 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.716841 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.716843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:16.717609 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:16.717611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.718015 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.718016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:16.718132 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:16.718599 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:16.718607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:16.719909 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:16.721129 (+ 1220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:16.721134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:16.721135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:16.721327 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:16.721501 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:16.721502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:16.721502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:16.721587 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:16.721781 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:16.721784 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:16.721880 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 19:00:16.721952 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:00:16.722053 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260925 19:00:16.722714 4417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42309
I20260925 19:00:16.722800 4417 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:16.722955 4417 heartbeater.cc:511] Master 127.0.98.62:42309 requested a full tablet report, sending...
I20260925 19:00:16.723449 3958 ts_manager.cc:194] Registered new tserver with Master: 4ea1cf12a1e74645a6e451ef23d191ec (127.0.98.3:43437)
I20260925 19:00:16.723656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 4288
I20260925 19:00:16.723763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:00:16.723958 392 external_mini_cluster.cc:1428] Setting key 2ef7a2c310899e1b49f6564d67371fa0
I20260925 19:00:16.723949 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54749
I20260925 19:00:16.724849 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 19:00:16.745911 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:16.738975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.0.98.1:33947)
0925 19:00:16.739155 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:16.739159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:16.739177 (+ 18us) server_negotiation.cc:407] Connection header received
0925 19:00:16.739203 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:16.739204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:16.739217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:16.739282 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:16.739782 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.740447 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.741093 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.741371 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.741420 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:16.743438 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:16.743445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:16.743445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:16.743451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:16.744991 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.745308 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.745310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.745311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.745365 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.745518 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.745518 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.745518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.745624 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:16.745725 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:16.745777 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 19:00:16.745836 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:16.752807 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:16.747099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.0.98.62:42309)
0925 19:00:16.747139 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:16.747141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:16.747177 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:00:16.747224 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:16.747225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:16.747233 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:16.747293 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:16.747674 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.748098 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.748672 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:16.748834 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:16.748877 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:16.750934 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:16.750935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:16.750936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:16.750941 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:16.752018 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.752292 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.752293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.752293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.752333 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:16.752458 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:16.752458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:16.752458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:16.752557 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:16.752613 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:16.752686 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 19:00:16.752751 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 19:00:16.752990 3958 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49694
I20260925 19:00:16.753259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 4020
I20260925 19:00:16.758008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 4154
I20260925 19:00:16.762125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 4288
I20260925 19:00:16.766462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 3926
2026-09-25T19:00:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1735 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[4429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[4429](info): set up 2 sockets
Sep 25 19:00:16 dist-test-slave-49bz krb5kdc[4429](info): commencing operation
krb5kdc: starting...
W20260925 19:00:17.762090 392 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:17 dist-test-slave-49bz krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:00:17.871145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:17.948906 4442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:17.949146 4442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:17.949183 4442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:17.950358 4442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:17.950423 4442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:17.950465 4442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:17.950500 4442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:17.950526 4442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:17.951944 4442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:17.952208 4442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:17.952414 4442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:17.956900 4448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:17.956900 4450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:17.957039 4442 server_base.cc:1034] running on GCE node
W20260925 19:00:17.956941 4447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:17.957366 4442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:17.957638 4442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:17.958803 4442 hybrid_clock.cc:648] HybridClock initialized: now 1790362817958798 us; error 38 us; skew 500 ppm
Sep 25 19:00:17 dist-test-slave-49bz krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:17.961133 4442 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:00:17.961434 4442 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:17.961501 4442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:17.961637 4442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:17.962653 4442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61"
format_stamp: "Formatted at 2026-09-25 19:00:17 on dist-test-slave-49bz"
server_key: "0c4908ac8ef17c02180f9fff9c3393f1"
server_key_iv: "1a1a3eb7ed531b07a48b05f48bf30a18"
server_key_version: "encryptionkey@0"
I20260925 19:00:17.962975 4442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61"
format_stamp: "Formatted at 2026-09-25 19:00:17 on dist-test-slave-49bz"
server_key: "0c4908ac8ef17c02180f9fff9c3393f1"
server_key_iv: "1a1a3eb7ed531b07a48b05f48bf30a18"
server_key_version: "encryptionkey@0"
I20260925 19:00:17.964148 4442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:00:17.964812 4456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:17.965027 4442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:17.965134 4442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61"
format_stamp: "Formatted at 2026-09-25 19:00:17 on dist-test-slave-49bz"
server_key: "0c4908ac8ef17c02180f9fff9c3393f1"
server_key_iv: "1a1a3eb7ed531b07a48b05f48bf30a18"
server_key_version: "encryptionkey@0"
I20260925 19:00:17.965215 4442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:17.983829 4442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:17.986042 4442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:17.986197 4442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:17.989658 4442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38259
I20260925 19:00:17.989704 4508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38259 every 8 connection(s)
I20260925 19:00:17.989988 4442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:17.990837 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:17.992965 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61: Bootstrap starting.
I20260925 19:00:17.993537 4509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:17.993817 4509 log.cc:824] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:17.994477 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61: No bootstrap required, opened a new log
I20260925 19:00:17.994872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 4442
I20260925 19:00:17.994978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:17.995172 392 external_mini_cluster.cc:1428] Setting key 26632286a4db56283225b5d5b619b9db
I20260925 19:00:17.997206 4509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } }
I20260925 19:00:17.997359 4509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:17.997390 4509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5faba436c84e4a3dae3d6a8fe8e0ce61, State: Initialized, Role: FOLLOWER
I20260925 19:00:17.997496 4509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } }
I20260925 19:00:17.997568 4509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:17.997596 4509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:17.997635 4509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:17.998270 4509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } }
I20260925 19:00:17.998365 4509 leader_election.cc:302] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5faba436c84e4a3dae3d6a8fe8e0ce61; no voters:
I20260925 19:00:17.998507 4509 leader_election.cc:288] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:17.998560 4514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:17.998713 4514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [term 1 LEADER]: Becoming Leader. State: Replica: 5faba436c84e4a3dae3d6a8fe8e0ce61, State: Running, Role: LEADER
I20260925 19:00:17.998757 4509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:17.998826 4514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } }
I20260925 19:00:17.999172 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5faba436c84e4a3dae3d6a8fe8e0ce61. Latest consensus state: current_term: 1 leader_uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } } }
I20260925 19:00:17.999423 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:17.999353 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5faba436c84e4a3dae3d6a8fe8e0ce61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } } }
I20260925 19:00:17.999794 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:17.999917 4523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:18.000449 4523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:18.001892 4523 catalog_manager.cc:1347] Generated new cluster ID: d2ce8e029c5e44b6a03784a251ff1aa8
I20260925 19:00:18.001953 4523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:00:18 dist-test-slave-49bz krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:18.007402 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:17.995666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.0.98.62:38259)
0925 19:00:17.995866 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:17.995875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:17.995907 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:00:17.996343 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:17.996351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:17.996411 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:17.996572 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:17.997099 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:17.998400 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:17.999577 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.000323 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.000586 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.003414 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:18.003418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:18.003418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:18.003429 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:18.005432 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:18.005819 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:18.005822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:18.005822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:18.005918 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:18.006110 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:18.006111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:18.006111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:18.006430 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:18.006572 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.007070 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.007215 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260925 19:00:18.007658 4523 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:18.008118 4523 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:18.016105 4523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61: Generated new TSK 0
I20260925 19:00:18.016273 4523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:00:18.042084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38259
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:18.118613 4535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:18.118804 4535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:18.118827 4535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:18.119973 4535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:18.120013 4535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:18.120064 4535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:18.121634 4535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:18.121884 4535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:18.122114 4535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:18.126580 4541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:18.126570 4540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:18.126572 4543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:18.126924 4535 server_base.cc:1034] running on GCE node
I20260925 19:00:18.127146 4535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:18.127390 4535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:18.128553 4535 hybrid_clock.cc:648] HybridClock initialized: now 1790362818128543 us; error 39 us; skew 500 ppm
Sep 25 19:00:18 dist-test-slave-49bz krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:18.130940 4535 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:00:18.131340 4535 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:18.131412 4535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:18.131554 4535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:18.132558 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f56648e887547b7aa85066e9f861726"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "069e2a457e73e990da276c6bf4abbf3d"
server_key_iv: "99246776649cbca54873404483a6f7a4"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.132900 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f56648e887547b7aa85066e9f861726"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "069e2a457e73e990da276c6bf4abbf3d"
server_key_iv: "99246776649cbca54873404483a6f7a4"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.134091 4535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:18.134883 4549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.135048 4535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:18.135139 4535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "8f56648e887547b7aa85066e9f861726"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "069e2a457e73e990da276c6bf4abbf3d"
server_key_iv: "99246776649cbca54873404483a6f7a4"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.135236 4535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:18.158923 4535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:18.160643 4535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:18.160792 4535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:18.160992 4535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:18.161276 4535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:18.161329 4535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.161367 4535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:18.161394 4535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.167176 4535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42587
I20260925 19:00:18.167201 4662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42587 every 8 connection(s)
I20260925 19:00:18.167462 4535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:00:18 dist-test-slave-49bz krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:18.176067 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.168104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54569 (local address 127.0.98.62:38259)
0925 19:00:18.168163 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.168165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.168349 (+ 184us) server_negotiation.cc:407] Connection header received
0925 19:00:18.168893 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.168894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.168904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.168950 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:18.169811 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.170316 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.171228 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.171404 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.171451 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.173286 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:18.173288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:18.173289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:18.173295 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:18.174738 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:18.175137 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:18.175139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:18.175139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:18.175195 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:18.175442 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:18.175443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:18.175443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:18.175549 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:18.175621 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.175883 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.175964 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:00:18.176231 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.168081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38259 (local address 127.0.98.1:54569)
0925 19:00:18.168232 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.168270 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:18.168818 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:18.169016 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:18.169019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:18.169276 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:18.169739 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.169743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.170386 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.170387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.171148 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.171150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.171470 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.171471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.171560 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.171964 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:18.171971 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:18.173192 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:18.174837 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:18.174843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:18.174845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:18.175051 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:18.175272 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:18.175273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:18.175273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:18.175373 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:18.175613 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:18.175618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:18.175754 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 19:00:18.175865 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 19:00:18.176016 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260925 19:00:18.176390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 4535
I20260925 19:00:18.176527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:18.176734 392 external_mini_cluster.cc:1428] Setting key 2cb4006f5459c3baf00d4641de819517
I20260925 19:00:18.176836 4663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38259
I20260925 19:00:18.176934 4663 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:18.177083 4663 heartbeater.cc:511] Master 127.0.98.62:38259 requested a full tablet report, sending...
I20260925 19:00:18.177632 4473 ts_manager.cc:194] Registered new tserver with Master: 8f56648e887547b7aa85066e9f861726 (127.0.98.1:42587)
I20260925 19:00:18.178555 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54569
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:00:18.200619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38259
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:18.279249 4668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:18.279465 4668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:18.279500 4668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:18.280625 4668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:18.280694 4668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:18.280737 4668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:18.282104 4668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:18.282346 4668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:18.282546 4668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:18.287113 4673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:18.287222 4676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:18.287223 4668 server_base.cc:1034] running on GCE node
W20260925 19:00:18.287222 4674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:18.287550 4668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:18.287775 4668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:18.288937 4668 hybrid_clock.cc:648] HybridClock initialized: now 1790362818288915 us; error 38 us; skew 500 ppm
Sep 25 19:00:18 dist-test-slave-49bz krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:18.291391 4668 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:00:18.291735 4668 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:18.291806 4668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:18.291945 4668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:18.292922 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "c19b0e3ddbf143dfab63218c1c4aad2b"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "d0e87077dd6279421a220685f1c93858"
server_key_iv: "8fa80a95b4286c067597c8defe562e3d"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.293257 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c19b0e3ddbf143dfab63218c1c4aad2b"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "d0e87077dd6279421a220685f1c93858"
server_key_iv: "8fa80a95b4286c067597c8defe562e3d"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.294480 4668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:00:18.295125 4682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.295279 4668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:18.295372 4668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "c19b0e3ddbf143dfab63218c1c4aad2b"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "d0e87077dd6279421a220685f1c93858"
server_key_iv: "8fa80a95b4286c067597c8defe562e3d"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.295451 4668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:18.308728 4668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:18.310899 4668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:18.311045 4668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:18.311245 4668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:18.311548 4668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:18.311599 4668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.311635 4668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:18.311663 4668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.317122 4668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38865
I20260925 19:00:18.317152 4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38865 every 8 connection(s)
I20260925 19:00:18.317432 4668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:00:18 dist-test-slave-49bz krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:18.325042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 4668
I20260925 19:00:18.325171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:00:18.325371 392 external_mini_cluster.cc:1428] Setting key fac25a5df748536830082cafdbe31272
I20260925 19:00:18.326174 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.318115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35063 (local address 127.0.98.62:38259)
0925 19:00:18.318195 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.318197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.318388 (+ 191us) server_negotiation.cc:407] Connection header received
0925 19:00:18.318874 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.318875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.318888 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.318931 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:18.319837 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.320334 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.321299 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.321502 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.321583 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.323435 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:18.323437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:18.323437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:18.323450 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:18.324779 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:18.325189 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:18.325190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:18.325190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:18.325260 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:18.325516 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:18.325517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:18.325517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:18.325673 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:18.325727 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.326022 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.326090 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:00:18.326403 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.318074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38259 (local address 127.0.98.2:35063)
0925 19:00:18.318239 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.318298 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:18.318771 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:18.319002 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:18.319006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:18.319284 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:18.319741 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.319746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.320434 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.320435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.321200 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.321202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.321591 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.321593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.321712 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.322093 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:18.322100 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:18.323328 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:18.324897 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:18.324902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:18.324904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:18.325105 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:18.325333 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:18.325334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:18.325334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:18.325452 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:18.325728 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:18.325734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:18.325898 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 19:00:18.326004 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 19:00:18.326139 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260925 19:00:18.326815 4796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38259
I20260925 19:00:18.326908 4796 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:18.327051 4796 heartbeater.cc:511] Master 127.0.98.62:38259 requested a full tablet report, sending...
I20260925 19:00:18.327371 4473 ts_manager.cc:194] Registered new tserver with Master: c19b0e3ddbf143dfab63218c1c4aad2b (127.0.98.2:38865)
I20260925 19:00:18.327724 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35063
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:00:18.348505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38259
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:18.424778 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:18.424973 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:18.424995 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:18.426034 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:18.426067 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:18.426091 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:18.427539 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:18.427709 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:18.427865 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:18.432340 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:18.432331 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:18.432405 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:18.432600 4801 server_base.cc:1034] running on GCE node
I20260925 19:00:18.432808 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:18.432984 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:18.434151 4801 hybrid_clock.cc:648] HybridClock initialized: now 1790362818434127 us; error 41 us; skew 500 ppm
Sep 25 19:00:18 dist-test-slave-49bz krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:18.436618 4801 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:00:18.436945 4801 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:18.437014 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:18.437158 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:18.438210 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "3c059df4b775424989a4912d2aa24149"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "4d2ff234a40935b9334b929034fa534c"
server_key_iv: "ed200cda50dd71f46659477160ad01da"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.438591 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3c059df4b775424989a4912d2aa24149"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "4d2ff234a40935b9334b929034fa534c"
server_key_iv: "ed200cda50dd71f46659477160ad01da"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.439801 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:18.440440 4815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.440595 4801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:18.440696 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "3c059df4b775424989a4912d2aa24149"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "4d2ff234a40935b9334b929034fa534c"
server_key_iv: "ed200cda50dd71f46659477160ad01da"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.440764 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:18.472573 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:18.474843 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:18.474995 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:18.475208 4801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:18.475507 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:18.475562 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.475602 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:18.475629 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.480998 4801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33099
I20260925 19:00:18.481031 4928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33099 every 8 connection(s)
I20260925 19:00:18.481335 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:00:18.482539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 4801
I20260925 19:00:18.482620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:00:18.482865 392 external_mini_cluster.cc:1428] Setting key 6705d81e8e231f931961b8ba1ed07966
Sep 25 19:00:18 dist-test-slave-49bz krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:18.490056 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.481994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39871 (local address 127.0.98.62:38259)
0925 19:00:18.482036 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.482038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.482224 (+ 186us) server_negotiation.cc:407] Connection header received
0925 19:00:18.482837 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.482838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.482850 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.482909 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:18.484074 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.484542 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.485433 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.485614 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.485672 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.487641 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:18.487643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:18.487643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:18.487648 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:18.488992 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:18.489330 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:18.489331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:18.489331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:18.489379 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:18.489578 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:18.489578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:18.489578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:18.489678 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:18.489724 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.489942 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.489996 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:00:18.490242 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.481949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38259 (local address 127.0.98.3:39871)
0925 19:00:18.482100 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.482138 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:18.482708 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:18.483032 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:18.483035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:18.483420 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:18.483998 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.484003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.484614 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.484615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.485364 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.485366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.485682 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.485682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.485774 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.486180 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:18.486187 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:18.487568 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:18.489059 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:18.489063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:18.489064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:18.489263 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:18.489436 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:18.489436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:18.489436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:18.489525 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:18.489731 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:18.489735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:18.489841 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 19:00:18.489922 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 19:00:18.490026 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260925 19:00:18.490674 4929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38259
I20260925 19:00:18.490751 4929 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:18.490859 4929 heartbeater.cc:511] Master 127.0.98.62:38259 requested a full tablet report, sending...
I20260925 19:00:18.491122 4473 ts_manager.cc:194] Registered new tserver with Master: 3c059df4b775424989a4912d2aa24149 (127.0.98.3:33099)
I20260925 19:00:18.491425 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39871
I20260925 19:00:18.494690 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:00:18.502013 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.495899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.0.98.62:38259)
0925 19:00:18.495971 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.495974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.496077 (+ 103us) server_negotiation.cc:407] Connection header received
0925 19:00:18.496148 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.496149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.496158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.496204 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:18.496744 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.497179 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.498003 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.498334 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.498394 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.499656 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:18.499663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:18.499663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:18.499670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:18.500940 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:18.501270 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:18.501270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:18.501271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:18.501320 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:18.501516 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:18.501517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:18.501517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:18.501646 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:18.501750 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.501853 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.501923 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:00:18.503288 4473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:00:18.504880 4473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:18.511310 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.507143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.0.98.2:38865)
0925 19:00:18.507329 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.507333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.507363 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:00:18.507410 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.507412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.507429 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.507506 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:18.507715 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.508771 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.510065 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.510816 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.510889 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.510943 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.511187 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.511230 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260925 19:00:18.511317 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.506923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38865 (local address 127.0.0.1:60480)
0925 19:00:18.507303 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.507313 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:18.507348 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:18.507586 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:18.507587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:18.507588 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:18.507630 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.507631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.508869 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.508870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.509973 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.509975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.510980 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.510982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.511068 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.511083 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 19:00:18.511160 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:00:18.511222 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":14,"thread_start_us":67,"threads_started":1}
I20260925 19:00:18.511592 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.507220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.0.98.1:42587)
0925 19:00:18.507421 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.507425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.507463 (+ 38us) server_negotiation.cc:407] Connection header received
0925 19:00:18.507586 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.507589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.507604 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.507687 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:18.507896 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.508558 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.510061 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.510815 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.510889 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.510943 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.511448 (+ 505us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.511492 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
I20260925 19:00:18.511317 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.506946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42587 (local address 127.0.0.1:39996)
0925 19:00:18.507417 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.507424 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:18.507472 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:18.507775 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:18.507776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:18.507778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:18.507820 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.507822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.508656 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.508657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.509974 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.509975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.510982 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.510983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.511068 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.511083 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 19:00:18.511160 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:00:18.511230 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":40,"thread_start_us":52,"threads_started":1}
I20260925 19:00:18.512111 4730 tablet_service.cc:1467] Processing CreateTablet for tablet 2410533944c0488db49555a0898590d8 (DEFAULT_TABLE table=test-table [id=c71456526a4c46c69f72afc97fd5f3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:18.512476 4730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2410533944c0488db49555a0898590d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:18.512421 4597 tablet_service.cc:1467] Processing CreateTablet for tablet 2410533944c0488db49555a0898590d8 (DEFAULT_TABLE table=test-table [id=c71456526a4c46c69f72afc97fd5f3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:18.512705 4597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2410533944c0488db49555a0898590d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:18.515719 4944 tablet_bootstrap.cc:492] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b: Bootstrap starting.
I20260925 19:00:18.516000 4943 tablet_bootstrap.cc:492] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726: Bootstrap starting.
I20260925 19:00:18.516057 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.506916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33099 (local address 127.0.0.1:45216)
0925 19:00:18.507139 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.507190 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:18.507253 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:18.507527 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:18.507530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:18.507533 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:18.507589 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.507591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.510075 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.510075 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.510805 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.510807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.515796 (+ 4989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.515798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.515876 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.515880 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:00:18.515950 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 19:00:18.515992 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":54,"threads_started":1}
I20260925 19:00:18.516074 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.506986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.0.98.3:33099)
0925 19:00:18.507377 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.507381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.507391 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:00:18.507426 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.507428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.507444 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.507517 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:18.507674 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.508770 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.510890 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.511509 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.511575 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.511628 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.515966 (+ 4338us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.516004 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":70,"threads_started":1}
I20260925 19:00:18.516521 4944 tablet_bootstrap.cc:654] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:18.516768 4863 tablet_service.cc:1467] Processing CreateTablet for tablet 2410533944c0488db49555a0898590d8 (DEFAULT_TABLE table=test-table [id=c71456526a4c46c69f72afc97fd5f3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:18.517031 4863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2410533944c0488db49555a0898590d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:18.517107 4944 log.cc:824] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:18.517174 4943 tablet_bootstrap.cc:654] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:18.517696 4943 log.cc:824] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:18.517874 4944 tablet_bootstrap.cc:492] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b: No bootstrap required, opened a new log
I20260925 19:00:18.517951 4944 ts_tablet_manager.cc:1397] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:00:18.518457 4943 tablet_bootstrap.cc:492] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726: No bootstrap required, opened a new log
I20260925 19:00:18.518532 4943 ts_tablet_manager.cc:1397] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:00:18.519840 4943 raft_consensus.cc:348] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } } peers { permanent_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38865 } } peers { permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } }
I20260925 19:00:18.519989 4943 raft_consensus.cc:374] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:18.520032 4943 raft_consensus.cc:729] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56648e887547b7aa85066e9f861726, State: Initialized, Role: FOLLOWER
I20260925 19:00:18.519989 4944 raft_consensus.cc:348] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } } peers { permanent_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38865 } } peers { permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } }
I20260925 19:00:18.520152 4944 raft_consensus.cc:374] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:18.520200 4944 raft_consensus.cc:729] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19b0e3ddbf143dfab63218c1c4aad2b, State: Initialized, Role: FOLLOWER
I20260925 19:00:18.520272 4943 consensus_queue.cc:260] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } } peers { permanent_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38865 } } peers { permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } }
I20260925 19:00:18.520311 4944 consensus_queue.cc:260] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } } peers { permanent_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38865 } } peers { permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } }
I20260925 19:00:18.520591 4943 ts_tablet_manager.cc:1428] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:00:18.520612 4944 ts_tablet_manager.cc:1428] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:00:18.520609 4663 heartbeater.cc:503] Master 127.0.98.62:38259 was elected leader, sending a full tablet report...
I20260925 19:00:18.520764 4796 heartbeater.cc:503] Master 127.0.98.62:38259 was elected leader, sending a full tablet report...
I20260925 19:00:18.521119 4947 tablet_bootstrap.cc:492] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: Bootstrap starting.
I20260925 19:00:18.521857 4947 tablet_bootstrap.cc:654] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:18.522408 4947 log.cc:824] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:18.522936 4947 tablet_bootstrap.cc:492] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: No bootstrap required, opened a new log
I20260925 19:00:18.522984 4947 ts_tablet_manager.cc:1397] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:18.524312 4947 raft_consensus.cc:348] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } } peers { permanent_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38865 } } peers { permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } }
I20260925 19:00:18.524425 4947 raft_consensus.cc:374] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:18.524446 4947 raft_consensus.cc:729] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c059df4b775424989a4912d2aa24149, State: Initialized, Role: FOLLOWER
I20260925 19:00:18.524515 4947 consensus_queue.cc:260] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } } peers { permanent_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38865 } } peers { permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } }
I20260925 19:00:18.524693 4947 ts_tablet_manager.cc:1428] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:00:18.524784 4929 heartbeater.cc:503] Master 127.0.98.62:38259 was elected leader, sending a full tablet report...
I20260925 19:00:18.530138 4949 raft_consensus.cc:482] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:18.530225 4949 raft_consensus.cc:504] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } } peers { permanent_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38865 } } peers { permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } }
I20260925 19:00:18.530545 4949 leader_election.cc:288] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c059df4b775424989a4912d2aa24149 (127.0.98.3:33099), 8f56648e887547b7aa85066e9f861726 (127.0.98.1:42587)
I20260925 19:00:18.533208 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.530775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38521 (local address 127.0.98.1:42587)
0925 19:00:18.530834 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.530836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.530897 (+ 61us) server_negotiation.cc:407] Connection header received
0925 19:00:18.530975 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.530975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.530986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.531042 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:18.531185 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.531637 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.532540 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.532874 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.532960 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.532967 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.533123 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.533153 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:00:18.533211 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.530697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42587 (local address 127.0.98.2:38521)
0925 19:00:18.530854 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.530864 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:18.530912 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:18.531045 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:18.531046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:18.531047 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:18.531104 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.531105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.531717 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.531718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.532440 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.532442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.532962 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.532963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.533035 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.533040 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:00:18.533115 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:00:18.533149 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260925 19:00:18.533269 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.530785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45377 (local address 127.0.98.3:33099)
0925 19:00:18.530823 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.530825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.530944 (+ 119us) server_negotiation.cc:407] Connection header received
0925 19:00:18.531015 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.531016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.531026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.531081 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:18.531255 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.531675 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.532589 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.532937 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.533016 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.533024 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.533176 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.533207 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:18.533375 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.530713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33099 (local address 127.0.98.2:45377)
0925 19:00:18.530894 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.530902 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:18.530949 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:18.531085 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:18.531086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:18.531087 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:18.531159 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.531161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.531769 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.531769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.532504 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:18.532506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.533015 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.533016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:18.533088 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.533091 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:18.533169 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 19:00:18.533206 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260925 19:00:18.533517 4617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2410533944c0488db49555a0898590d8" candidate_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56648e887547b7aa85066e9f861726" is_pre_election: true
I20260925 19:00:18.533633 4617 raft_consensus.cc:2454] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19b0e3ddbf143dfab63218c1c4aad2b in term 0.
I20260925 19:00:18.533659 4883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2410533944c0488db49555a0898590d8" candidate_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c059df4b775424989a4912d2aa24149" is_pre_election: true
I20260925 19:00:18.533774 4883 raft_consensus.cc:2454] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19b0e3ddbf143dfab63218c1c4aad2b in term 0.
I20260925 19:00:18.533849 4684 leader_election.cc:302] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56648e887547b7aa85066e9f861726, c19b0e3ddbf143dfab63218c1c4aad2b; no voters:
I20260925 19:00:18.533979 4949 raft_consensus.cc:2781] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:18.534029 4949 raft_consensus.cc:482] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:18.534044 4949 raft_consensus.cc:3037] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:18.534634 4949 raft_consensus.cc:504] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } } peers { permanent_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38865 } } peers { permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } }
I20260925 19:00:18.534761 4949 leader_election.cc:288] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [CANDIDATE]: Term 1 election: Requested vote from peers 3c059df4b775424989a4912d2aa24149 (127.0.98.3:33099), 8f56648e887547b7aa85066e9f861726 (127.0.98.1:42587)
I20260925 19:00:18.534894 4883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2410533944c0488db49555a0898590d8" candidate_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c059df4b775424989a4912d2aa24149"
I20260925 19:00:18.534894 4617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2410533944c0488db49555a0898590d8" candidate_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56648e887547b7aa85066e9f861726"
I20260925 19:00:18.534962 4617 raft_consensus.cc:3037] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:18.534960 4883 raft_consensus.cc:3037] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:18.535557 4883 raft_consensus.cc:2454] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19b0e3ddbf143dfab63218c1c4aad2b in term 1.
I20260925 19:00:18.535631 4617 raft_consensus.cc:2454] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19b0e3ddbf143dfab63218c1c4aad2b in term 1.
I20260925 19:00:18.535739 4683 leader_election.cc:302] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c059df4b775424989a4912d2aa24149, c19b0e3ddbf143dfab63218c1c4aad2b; no voters:
I20260925 19:00:18.535831 4949 raft_consensus.cc:2781] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:18.535966 4949 raft_consensus.cc:686] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 1 LEADER]: Becoming Leader. State: Replica: c19b0e3ddbf143dfab63218c1c4aad2b, State: Running, Role: LEADER
I20260925 19:00:18.536075 4949 consensus_queue.cc:237] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } } peers { permanent_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38865 } } peers { permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } }
I20260925 19:00:18.536684 4472 catalog_manager.cc:5495] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b reported cstate change: term changed from 0 to 1, leader changed from <none> to c19b0e3ddbf143dfab63218c1c4aad2b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c19b0e3ddbf143dfab63218c1c4aad2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20260925 19:00:18.548341 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.545938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.0.98.2:38865)
0925 19:00:18.545987 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.545989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.545998 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:18.546088 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.546088 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.546106 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.546161 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:18.546325 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.546798 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.547566 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:18.547763 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:18.547843 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:18.548013 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:18.548110 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:18.548200 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.548222 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.548240 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:00:18.549851 4883 raft_consensus.cc:1261] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149 [term 1 FOLLOWER]: Refusing update from remote peer c19b0e3ddbf143dfab63218c1c4aad2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:18.549880 4617 raft_consensus.cc:1261] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726 [term 1 FOLLOWER]: Refusing update from remote peer c19b0e3ddbf143dfab63218c1c4aad2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:18.550175 4949 consensus_queue.cc:1035] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c059df4b775424989a4912d2aa24149" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:18.550355 4953 consensus_queue.cc:1035] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56648e887547b7aa85066e9f861726" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:18.553311 4957 mvcc.cc:205] Tried to move back new op lower bound from 7333326104778006528 to 7333326104724279296. Current Snapshot: MvccSnapshot[applied={T|T < 7333326104778006528 or (T in {7333326104778006528})}]
I20260925 19:00:18.553427 4956 mvcc.cc:205] Tried to move back new op lower bound from 7333326104778006528 to 7333326104724279296. Current Snapshot: MvccSnapshot[applied={T|T < 7333326104778006528 or (T in {7333326104778006528})}]
I20260925 19:00:18.554646 4473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:18.554719 4473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:18.555387 4473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5faba436c84e4a3dae3d6a8fe8e0ce61: Sending DeleteTablet for 3 replicas of tablet 2410533944c0488db49555a0898590d8
I20260925 19:00:18.555684 4730 tablet_service.cc:1514] Processing DeleteTablet for tablet 2410533944c0488db49555a0898590d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60480
I20260925 19:00:18.555708 4597 tablet_service.cc:1514] Processing DeleteTablet for tablet 2410533944c0488db49555a0898590d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39996
I20260925 19:00:18.555913 4967 tablet_replica.cc:330] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726: stopping tablet replica
I20260925 19:00:18.556063 4967 raft_consensus.cc:2229] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:18.556073 4863 tablet_service.cc:1514] Processing DeleteTablet for tablet 2410533944c0488db49555a0898590d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45216
I20260925 19:00:18.556133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 4535
I20260925 19:00:18.556244 4968 tablet_replica.cc:330] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: stopping tablet replica
I20260925 19:00:18.556327 4968 raft_consensus.cc:2229] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:18.556327 4967 raft_consensus.cc:2258] T 2410533944c0488db49555a0898590d8 P 8f56648e887547b7aa85066e9f861726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:18.556414 4968 raft_consensus.cc:2258] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:18.556840 4968 ts_tablet_manager.cc:1905] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:18.558382 4968 ts_tablet_manager.cc:1918] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:18.558457 4968 log.cc:1196] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/2410533944c0488db49555a0898590d8
I20260925 19:00:18.558753 4968 ts_tablet_manager.cc:1939] T 2410533944c0488db49555a0898590d8 P 3c059df4b775424989a4912d2aa24149: Deleting consensus metadata
I20260925 19:00:18.559517 4457 catalog_manager.cc:4841] TS 3c059df4b775424989a4912d2aa24149 (127.0.98.3:33099): tablet 2410533944c0488db49555a0898590d8 (table test-table [id=c71456526a4c46c69f72afc97fd5f3c9]) successfully deleted
W20260925 19:00:18.561887 4458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42587 (error 108)
I20260925 19:00:18.562706 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.562571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42587 (local address 127.0.0.1:40008)
0925 19:00:18.562631 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.562674 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 19:00:18.562705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 4668
I20260925 19:00:18.562718 4966 tablet_replica.cc:330] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b: stopping tablet replica
W20260925 19:00:18.562825 4458 catalog_manager.cc:4584] TS 8f56648e887547b7aa85066e9f861726 (127.0.98.1:42587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2410533944c0488db49555a0898590d8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42587: connect: Connection refused (error 111)
I20260925 19:00:18.562969 4966 raft_consensus.cc:2229] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:18.563117 4966 raft_consensus.cc:2258] T 2410533944c0488db49555a0898590d8 P c19b0e3ddbf143dfab63218c1c4aad2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:18.566998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 4801
I20260925 19:00:18.567036 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.566924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38865 (local address 127.0.0.1:60490)
0925 19:00:18.566976 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.567007 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 19:00:18.571625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 4442
2026-09-25T19:00:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1805 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T19:00:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:18Z Disabled control of system clock
I20260925 19:00:18.597786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42535
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 19:00:18.673606 4973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:18.673784 4973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:18.673803 4973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:18.674822 4973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:18.674856 4973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:18.674868 4973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:18.674880 4973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:18.674891 4973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:18.676049 4973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:18.676203 4973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:18.676406 4973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:18.680796 4981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:18.680785 4978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:18.680989 4979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:18.681113 4973 server_base.cc:1034] running on GCE node
I20260925 19:00:18.681339 4973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:18.681597 4973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:18.682736 4973 hybrid_clock.cc:648] HybridClock initialized: now 1790362818682729 us; error 30 us; skew 500 ppm
I20260925 19:00:18.683928 4973 webserver.cc:460] Webserver started at http://127.0.98.62:37533/ using document root <none> and password file <none>
I20260925 19:00:18.684154 4973 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:18.684195 4973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:18.684348 4973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:18.685312 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "7f59cafaee6842ff9c7fca5b912dbcc1"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "6cec0c79062230292d866344f19e5c78"
server_key_iv: "34d2e675bc32825429578a2eaba2d6a8"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.685655 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f59cafaee6842ff9c7fca5b912dbcc1"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "6cec0c79062230292d866344f19e5c78"
server_key_iv: "34d2e675bc32825429578a2eaba2d6a8"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.686904 4973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:00:18.687589 4987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.687755 4973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:18.687835 4973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "7f59cafaee6842ff9c7fca5b912dbcc1"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "6cec0c79062230292d866344f19e5c78"
server_key_iv: "34d2e675bc32825429578a2eaba2d6a8"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.687906 4973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:18.691002 4973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:18.691259 4973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:18.691393 4973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:18.695176 4973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42535
I20260925 19:00:18.695202 5039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42535 every 8 connection(s)
I20260925 19:00:18.695489 4973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:18.696036 5040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:18.698513 5040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1: Bootstrap starting.
I20260925 19:00:18.699035 5040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:18.699357 5040 log.cc:824] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:18.699965 5040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1: No bootstrap required, opened a new log
I20260925 19:00:18.701423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 4973
I20260925 19:00:18.701498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:18.701510 5040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f59cafaee6842ff9c7fca5b912dbcc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42535 } }
I20260925 19:00:18.701661 392 external_mini_cluster.cc:1428] Setting key 46c626532c081a0307ac496edbb47652
I20260925 19:00:18.701668 5040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:18.701712 5040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f59cafaee6842ff9c7fca5b912dbcc1, State: Initialized, Role: FOLLOWER
I20260925 19:00:18.701882 5040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f59cafaee6842ff9c7fca5b912dbcc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42535 } }
I20260925 19:00:18.701973 5040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:18.702008 5040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:18.702049 5040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:18.702760 5040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f59cafaee6842ff9c7fca5b912dbcc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42535 } }
I20260925 19:00:18.702858 5040 leader_election.cc:302] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f59cafaee6842ff9c7fca5b912dbcc1; no voters:
I20260925 19:00:18.703302 5040 leader_election.cc:288] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:18.703394 5045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:18.703553 5045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [term 1 LEADER]: Becoming Leader. State: Replica: 7f59cafaee6842ff9c7fca5b912dbcc1, State: Running, Role: LEADER
I20260925 19:00:18.703583 5040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:18.703639 5045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f59cafaee6842ff9c7fca5b912dbcc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42535 } }
I20260925 19:00:18.703897 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f59cafaee6842ff9c7fca5b912dbcc1. Latest consensus state: current_term: 1 leader_uuid: "7f59cafaee6842ff9c7fca5b912dbcc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f59cafaee6842ff9c7fca5b912dbcc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42535 } } }
I20260925 19:00:18.703971 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:18.704026 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.702136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.0.98.62:42535)
0925 19:00:18.702362 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.702370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.702426 (+ 56us) server_negotiation.cc:407] Connection header received
0925 19:00:18.702578 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.702584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.702629 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.702773 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:18.703402 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:18.703406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:18.703407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:18.703411 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:18.703456 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:18.703469 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:18.703532 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.703746 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.703808 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260925 19:00:18.704352 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f59cafaee6842ff9c7fca5b912dbcc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f59cafaee6842ff9c7fca5b912dbcc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42535 } } }
I20260925 19:00:18.704425 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1 [sys.catalog]: This master's current role is: LEADER
W20260925 19:00:18.705293 5060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:00:18.705341 5060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:00:18.705421 5051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:18.705560 5051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:18.706825 5051 catalog_manager.cc:1347] Generated new cluster ID: d80b389fbec84d799383763bb0334908
I20260925 19:00:18.706871 5051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:18.729555 5051 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:18.729712 5051 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:18.735174 5051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1: Generated new TSK 0
I20260925 19:00:18.735342 5051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:00:18.736421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42535
--builtin_ntp_servers=127.0.98.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 19:00:18.814349 5064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:18.814527 5064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:18.814548 5064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:18.815699 5064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:18.815738 5064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:18.815763 5064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:18.817040 5064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:18.817276 5064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:18.817556 5064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:18.822329 5069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:18.822376 5070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:18.822329 5072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:18.822832 5064 server_base.cc:1034] running on GCE node
I20260925 19:00:18.823042 5064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:18.823304 5064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:18.824460 5064 hybrid_clock.cc:648] HybridClock initialized: now 1790362818824446 us; error 26 us; skew 500 ppm
I20260925 19:00:18.825706 5064 webserver.cc:460] Webserver started at http://127.0.98.1:45653/ using document root <none> and password file <none>
I20260925 19:00:18.825974 5064 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:18.826042 5064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:18.826188 5064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:18.827191 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "36eede2a761ca344149c46ba472d331a"
server_key_iv: "124ba640ea63bd1d8452327cdb97059b"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.827535 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "36eede2a761ca344149c46ba472d331a"
server_key_iv: "124ba640ea63bd1d8452327cdb97059b"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.828822 5064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:00:18.829484 5078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.829656 5064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:18.829751 5064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "36eede2a761ca344149c46ba472d331a"
server_key_iv: "124ba640ea63bd1d8452327cdb97059b"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.829833 5064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:18.833141 5064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:18.833396 5064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:18.833530 5064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:18.833731 5064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:18.834017 5064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:18.834070 5064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.834120 5064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:18.834146 5064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.839754 5064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40727
I20260925 19:00:18.839840 5191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40727 every 8 connection(s)
I20260925 19:00:18.840123 5064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:00:18.841130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 5064
I20260925 19:00:18.841239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:18.841404 392 external_mini_cluster.cc:1428] Setting key 1cc4f4005c36896e3eb66c906d071930
I20260925 19:00:18.842216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42535
--builtin_ntp_servers=127.0.98.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 19:00:18.843118 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.840826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55839 (local address 127.0.98.62:42535)
0925 19:00:18.840865 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.840867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.841062 (+ 195us) server_negotiation.cc:407] Connection header received
0925 19:00:18.841563 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.841564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.841571 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.841645 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:18.842856 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:18.842858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:18.842858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:18.842863 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:18.842881 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:18.842886 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:18.842952 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.843003 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.843020 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:00:18.843227 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.840801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42535 (local address 127.0.98.1:55839)
0925 19:00:18.840950 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.840978 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:18.841444 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:18.841685 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:18.841689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:18.841698 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:18.842710 (+ 1012us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:00:18.842711 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:18.842742 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:00:18.842769 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:18.842950 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:18.842952 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:18.842954 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:00:18.842997 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 19:00:18.843038 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260925 19:00:18.843757 5192 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42535
I20260925 19:00:18.843854 5192 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:18.843984 5192 heartbeater.cc:511] Master 127.0.98.62:42535 requested a full tablet report, sending...
I20260925 19:00:18.844457 5004 ts_manager.cc:194] Registered new tserver with Master: faa6a63220ac4b1eaed2770ec1e6c1fc (127.0.98.1:40727)
W20260925 19:00:18.921133 5195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:18.921314 5195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:18.921334 5195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:18.922446 5195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:18.922480 5195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:18.922505 5195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:18.923653 5195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:18.923837 5195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:18.924053 5195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:18.928730 5203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:18.928865 5195 server_base.cc:1034] running on GCE node
W20260925 19:00:18.928738 5201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:18.929039 5200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:18.929327 5195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:18.929565 5195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:18.930727 5195 hybrid_clock.cc:648] HybridClock initialized: now 1790362818930717 us; error 36 us; skew 500 ppm
I20260925 19:00:18.931831 5195 webserver.cc:460] Webserver started at http://127.0.98.2:43683/ using document root <none> and password file <none>
I20260925 19:00:18.932045 5195 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:18.932116 5195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:18.932320 5195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:18.933373 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "84019a09f0374b9aa6ef2751cae773b9"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "224c0533f5e839913850f799914eb8c4"
server_key_iv: "eb93afd93ad7df80c3fda50fb10e5040"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.933737 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84019a09f0374b9aa6ef2751cae773b9"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "224c0533f5e839913850f799914eb8c4"
server_key_iv: "eb93afd93ad7df80c3fda50fb10e5040"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.935061 5195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:00:18.935775 5209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.935961 5195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:18.936057 5195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "84019a09f0374b9aa6ef2751cae773b9"
format_stamp: "Formatted at 2026-09-25 19:00:18 on dist-test-slave-49bz"
server_key: "224c0533f5e839913850f799914eb8c4"
server_key_iv: "eb93afd93ad7df80c3fda50fb10e5040"
server_key_version: "encryptionkey@0"
I20260925 19:00:18.936142 5195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:18.939375 5195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:18.939625 5195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:18.939759 5195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:18.939970 5195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:18.940255 5195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:18.940308 5195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.940346 5195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:18.940372 5195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:18.945901 5195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32959
I20260925 19:00:18.945950 5322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32959 every 8 connection(s)
I20260925 19:00:18.946242 5195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:00:18.947082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 5195
I20260925 19:00:18.947170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:00:18.947348 392 external_mini_cluster.cc:1428] Setting key 08662f19dfc213bb127addb3bb6492ee
I20260925 19:00:18.948218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42535
--builtin_ntp_servers=127.0.98.20:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 19:00:18.949350 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.946961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49023 (local address 127.0.98.62:42535)
0925 19:00:18.947038 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:18.947040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:18.947245 (+ 205us) server_negotiation.cc:407] Connection header received
0925 19:00:18.947829 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:18.947830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:18.947836 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:18.947895 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:18.949095 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:18.949096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:18.949097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:18.949101 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:18.949116 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:18.949120 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:18.949170 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:18.949252 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 19:00:18.949265 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:00:18.949530 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:18.946912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42535 (local address 127.0.98.2:49023)
0925 19:00:18.947124 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:18.947160 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:18.947709 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:18.947962 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:18.947966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:18.947975 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:18.948924 (+ 949us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:00:18.948925 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:18.948961 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:00:18.948986 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:18.949175 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:18.949179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:18.949182 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:18.949261 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:00:18.949334 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260925 19:00:18.950012 5323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42535
I20260925 19:00:18.950109 5323 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:18.950274 5323 heartbeater.cc:511] Master 127.0.98.62:42535 requested a full tablet report, sending...
I20260925 19:00:18.950629 5004 ts_manager.cc:194] Registered new tserver with Master: 84019a09f0374b9aa6ef2751cae773b9 (127.0.98.2:32959)
W20260925 19:00:19.025317 5326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:19.025547 5326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:19.025580 5326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:19.026726 5326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:19.026779 5326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:19.026819 5326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:19.028047 5326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:19.028309 5326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:19.028581 5326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:19.033344 5331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:19.033361 5334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:19.033370 5332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:19.033568 5326 server_base.cc:1034] running on GCE node
I20260925 19:00:19.033756 5326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:19.034013 5326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:19.035145 5326 hybrid_clock.cc:648] HybridClock initialized: now 1790362819035136 us; error 26 us; skew 500 ppm
I20260925 19:00:19.036314 5326 webserver.cc:460] Webserver started at http://127.0.98.3:39901/ using document root <none> and password file <none>
I20260925 19:00:19.036559 5326 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:19.036604 5326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:19.036738 5326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:19.037703 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "44a05166bf704d85ac164b9c1693e59c"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "2280c80f51b1267fb8a52390865ae237"
server_key_iv: "184e10b0f155612d83c701a9add0a779"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.038053 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "44a05166bf704d85ac164b9c1693e59c"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "2280c80f51b1267fb8a52390865ae237"
server_key_iv: "184e10b0f155612d83c701a9add0a779"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.039443 5326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:19.040139 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:19.040309 5326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:19.040380 5326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "44a05166bf704d85ac164b9c1693e59c"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "2280c80f51b1267fb8a52390865ae237"
server_key_iv: "184e10b0f155612d83c701a9add0a779"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.040462 5326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:19.043581 5326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:19.043809 5326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:19.043934 5326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:19.044130 5326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:19.044399 5326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:19.044430 5326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:19.044482 5326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:19.044509 5326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:19.049906 5326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41191
I20260925 19:00:19.049964 5453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41191 every 8 connection(s)
I20260925 19:00:19.050238 5326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:00:19.052981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 5326
I20260925 19:00:19.053085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:00:19.053198 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.051011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52845 (local address 127.0.98.62:42535)
0925 19:00:19.051052 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.051054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.051322 (+ 268us) server_negotiation.cc:407] Connection header received
0925 19:00:19.051770 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.051771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.051777 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.051858 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.052917 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.052918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.052919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.052923 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.052936 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.052941 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.052991 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.053121 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.053137 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:19.053262 392 external_mini_cluster.cc:1428] Setting key 08aae2257b9b0c55928f09baac70c81d
I20260925 19:00:19.053404 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.050988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42535 (local address 127.0.98.3:52845)
0925 19:00:19.051197 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:19.051243 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:19.051673 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:19.051919 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:19.051923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:19.051932 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:19.052748 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:00:19.052748 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:19.052772 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:00:19.052822 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:19.053005 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:19.053009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:19.053012 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:19.053121 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 19:00:19.053183 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260925 19:00:19.053857 5454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42535
I20260925 19:00:19.053942 5454 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:19.054075 5454 heartbeater.cc:511] Master 127.0.98.62:42535 requested a full tablet report, sending...
I20260925 19:00:19.054383 5004 ts_manager.cc:194] Registered new tserver with Master: 44a05166bf704d85ac164b9c1693e59c (127.0.98.3:41191)
I20260925 19:00:19.064751 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:00:19.067057 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.065949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.0.98.62:42535)
0925 19:00:19.066020 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.066022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.066119 (+ 97us) server_negotiation.cc:407] Connection header received
0925 19:00:19.066184 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.066185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.066190 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.066266 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.066860 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.066861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.066862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.066866 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.066880 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.066884 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.066934 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.066982 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.066995 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:00:19.068379 5004 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:00:19.069788 5004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:19.073333 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.071804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40727 (local address 127.0.0.1:33634)
0925 19:00:19.071948 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:19.071978 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:19.072028 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:19.072244 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:19.072248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:19.072253 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:19.072971 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:00:19.072971 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:19.072976 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:00:19.072986 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:19.073185 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:19.073190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:19.073193 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:19.073259 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 19:00:19.073272 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260925 19:00:19.073366 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.071958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.0.98.1:40727)
0925 19:00:19.072134 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.072138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.072147 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:19.072172 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.072173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.072183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.072246 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.073067 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.073072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.073072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.073076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.073102 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.073117 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.073190 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.073267 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.073282 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260925 19:00:19.073577 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.071949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.0.98.3:41191)
0925 19:00:19.072226 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.072229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.072237 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:19.072257 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.072260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.072270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.072373 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.073263 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.073266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.073267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.073270 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.073299 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.073312 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.073368 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.073498 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.073511 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":59,"threads_started":1}
I20260925 19:00:19.073722 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.072036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.0.98.2:32959)
0925 19:00:19.072184 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.072188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.072679 (+ 491us) server_negotiation.cc:407] Connection header received
0925 19:00:19.072708 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.072710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.072719 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.072789 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.073259 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.073266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.073267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.073270 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.073299 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.073319 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.073368 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.073620 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.073635 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260925 19:00:19.073553 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.071840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32959 (local address 127.0.0.1:37818)
0925 19:00:19.072568 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:19.072578 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:19.072621 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:19.072807 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:19.072808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:19.072811 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:19.073149 (+ 338us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:00:19.073149 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:19.073156 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:00:19.073170 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:19.073394 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:19.073395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:19.073396 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:00:19.073451 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 19:00:19.073467 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":81,"threads_started":1}
I20260925 19:00:19.073555 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.071804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41191 (local address 127.0.0.1:52568)
0925 19:00:19.072105 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:19.072114 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:19.072144 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:19.072375 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:19.072376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:19.072379 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:19.072796 (+ 417us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:00:19.072797 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:19.072810 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:00:19.072830 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:19.073394 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:19.073395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:19.073396 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:00:19.073454 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:00:19.073468 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":5,"thread_start_us":50,"threads_started":1}
I20260925 19:00:19.075026 5126 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2248cd04174c98b1b2a65c9be34fc0 (DEFAULT_TABLE table=test-table [id=adac823aa06d44ef8509b23e2b305574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:19.075079 5257 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2248cd04174c98b1b2a65c9be34fc0 (DEFAULT_TABLE table=test-table [id=adac823aa06d44ef8509b23e2b305574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:19.075073 5388 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2248cd04174c98b1b2a65c9be34fc0 (DEFAULT_TABLE table=test-table [id=adac823aa06d44ef8509b23e2b305574]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:19.075273 5126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2248cd04174c98b1b2a65c9be34fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:19.075366 5388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2248cd04174c98b1b2a65c9be34fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:19.075371 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2248cd04174c98b1b2a65c9be34fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:19.077487 5468 tablet_bootstrap.cc:492] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc: Bootstrap starting.
I20260925 19:00:19.078307 5468 tablet_bootstrap.cc:654] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:19.078342 5470 tablet_bootstrap.cc:492] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: Bootstrap starting.
I20260925 19:00:19.078639 5468 log.cc:824] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:19.078998 5469 tablet_bootstrap.cc:492] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: Bootstrap starting.
I20260925 19:00:19.079082 5470 tablet_bootstrap.cc:654] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:19.079396 5468 tablet_bootstrap.cc:492] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc: No bootstrap required, opened a new log
I20260925 19:00:19.079420 5470 log.cc:824] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:19.079479 5468 ts_tablet_manager.cc:1397] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:00:19.079636 5469 tablet_bootstrap.cc:654] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:19.079943 5469 log.cc:824] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:19.080139 5470 tablet_bootstrap.cc:492] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: No bootstrap required, opened a new log
I20260925 19:00:19.080204 5470 ts_tablet_manager.cc:1397] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:19.080766 5469 tablet_bootstrap.cc:492] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: No bootstrap required, opened a new log
I20260925 19:00:19.080833 5469 ts_tablet_manager.cc:1397] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:19.081795 5468 raft_consensus.cc:348] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 } } peers { permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 } } peers { permanent_uuid: "84019a09f0374b9aa6ef2751cae773b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32959 } }
I20260925 19:00:19.081948 5468 raft_consensus.cc:374] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:19.081990 5468 raft_consensus.cc:729] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa6a63220ac4b1eaed2770ec1e6c1fc, State: Initialized, Role: FOLLOWER
I20260925 19:00:19.082096 5468 consensus_queue.cc:260] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 } } peers { permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 } } peers { permanent_uuid: "84019a09f0374b9aa6ef2751cae773b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32959 } }
I20260925 19:00:19.082357 5468 ts_tablet_manager.cc:1428] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:00:19.082491 5192 heartbeater.cc:503] Master 127.0.98.62:42535 was elected leader, sending a full tablet report...
I20260925 19:00:19.082636 5470 raft_consensus.cc:348] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 } } peers { permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 } } peers { permanent_uuid: "84019a09f0374b9aa6ef2751cae773b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32959 } }
I20260925 19:00:19.082813 5470 raft_consensus.cc:374] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:19.082859 5470 raft_consensus.cc:729] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84019a09f0374b9aa6ef2751cae773b9, State: Initialized, Role: FOLLOWER
I20260925 19:00:19.083016 5470 consensus_queue.cc:260] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 } } peers { permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 } } peers { permanent_uuid: "84019a09f0374b9aa6ef2751cae773b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32959 } }
I20260925 19:00:19.083261 5470 ts_tablet_manager.cc:1428] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 19:00:19.083280 5323 heartbeater.cc:503] Master 127.0.98.62:42535 was elected leader, sending a full tablet report...
I20260925 19:00:19.083359 5469 raft_consensus.cc:348] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 } } peers { permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 } } peers { permanent_uuid: "84019a09f0374b9aa6ef2751cae773b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32959 } }
I20260925 19:00:19.083516 5469 raft_consensus.cc:374] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:19.083554 5469 raft_consensus.cc:729] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a05166bf704d85ac164b9c1693e59c, State: Initialized, Role: FOLLOWER
I20260925 19:00:19.083662 5469 consensus_queue.cc:260] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 } } peers { permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 } } peers { permanent_uuid: "84019a09f0374b9aa6ef2751cae773b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32959 } }
I20260925 19:00:19.083905 5469 ts_tablet_manager.cc:1428] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:00:19.083968 5454 heartbeater.cc:503] Master 127.0.98.62:42535 was elected leader, sending a full tablet report...
W20260925 19:00:19.090579 5193 tablet.cc:2369] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:00:19.137281 5475 raft_consensus.cc:482] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:19.137384 5475 raft_consensus.cc:504] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 } } peers { permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 } } peers { permanent_uuid: "84019a09f0374b9aa6ef2751cae773b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32959 } }
I20260925 19:00:19.137756 5475 leader_election.cc:288] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44a05166bf704d85ac164b9c1693e59c (127.0.98.3:41191), faa6a63220ac4b1eaed2770ec1e6c1fc (127.0.98.1:40727)
I20260925 19:00:19.139039 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.137926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41191 (local address 127.0.98.2:35833)
0925 19:00:19.137988 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:19.138000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:19.138048 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:19.138215 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:19.138216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:19.138218 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:19.138708 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:00:19.138709 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:19.138717 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:00:19.138730 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:19.138924 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:19.138925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:19.138925 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 19:00:19.138983 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:00:19.138994 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 19:00:19.139256 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.138049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35833 (local address 127.0.98.3:41191)
0925 19:00:19.138103 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.138104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.138113 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:19.138140 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.138141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.138146 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.138208 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.138811 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.138815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.138815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.138823 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.138837 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.138855 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.138916 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.139151 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.139179 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:00:19.139269 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.138010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41687 (local address 127.0.98.1:40727)
0925 19:00:19.138048 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.138050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.138198 (+ 148us) server_negotiation.cc:407] Connection header received
0925 19:00:19.138279 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.138284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.138290 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.138352 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.138994 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.138996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.138996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.139000 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.139014 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.139018 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.139109 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.139200 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.139213 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:19.139465 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.137926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40727 (local address 127.0.98.2:41687)
0925 19:00:19.138113 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:19.138132 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:19.138199 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:19.138357 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:19.138358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:19.138372 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:19.138894 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:00:19.138895 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:19.138902 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:00:19.138915 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:19.139108 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:19.139109 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:19.139110 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:00:19.139363 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 19:00:19.139377 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":58,"thread_start_us":84,"threads_started":1}
I20260925 19:00:19.139597 5408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2248cd04174c98b1b2a65c9be34fc0" candidate_uuid: "84019a09f0374b9aa6ef2751cae773b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a05166bf704d85ac164b9c1693e59c" is_pre_election: true
I20260925 19:00:19.139775 5408 raft_consensus.cc:2454] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84019a09f0374b9aa6ef2751cae773b9 in term 0.
I20260925 19:00:19.139873 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2248cd04174c98b1b2a65c9be34fc0" candidate_uuid: "84019a09f0374b9aa6ef2751cae773b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" is_pre_election: true
I20260925 19:00:19.140017 5146 raft_consensus.cc:2454] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84019a09f0374b9aa6ef2751cae773b9 in term 0.
I20260925 19:00:19.140028 5213 leader_election.cc:302] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a05166bf704d85ac164b9c1693e59c, 84019a09f0374b9aa6ef2751cae773b9; no voters:
I20260925 19:00:19.140141 5475 raft_consensus.cc:2781] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:19.140197 5475 raft_consensus.cc:482] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:19.140223 5475 raft_consensus.cc:3037] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:19.140919 5475 raft_consensus.cc:504] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 } } peers { permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 } } peers { permanent_uuid: "84019a09f0374b9aa6ef2751cae773b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32959 } }
I20260925 19:00:19.141057 5475 leader_election.cc:288] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [CANDIDATE]: Term 1 election: Requested vote from peers 44a05166bf704d85ac164b9c1693e59c (127.0.98.3:41191), faa6a63220ac4b1eaed2770ec1e6c1fc (127.0.98.1:40727)
I20260925 19:00:19.141201 5408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2248cd04174c98b1b2a65c9be34fc0" candidate_uuid: "84019a09f0374b9aa6ef2751cae773b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a05166bf704d85ac164b9c1693e59c"
I20260925 19:00:19.141202 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2248cd04174c98b1b2a65c9be34fc0" candidate_uuid: "84019a09f0374b9aa6ef2751cae773b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc"
I20260925 19:00:19.141268 5408 raft_consensus.cc:3037] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:19.141268 5146 raft_consensus.cc:3037] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:19.142086 5146 raft_consensus.cc:2454] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84019a09f0374b9aa6ef2751cae773b9 in term 1.
I20260925 19:00:19.142086 5408 raft_consensus.cc:2454] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84019a09f0374b9aa6ef2751cae773b9 in term 1.
I20260925 19:00:19.142290 5213 leader_election.cc:302] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a05166bf704d85ac164b9c1693e59c, 84019a09f0374b9aa6ef2751cae773b9; no voters:
I20260925 19:00:19.142391 5475 raft_consensus.cc:2781] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:19.142540 5475 raft_consensus.cc:686] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 1 LEADER]: Becoming Leader. State: Replica: 84019a09f0374b9aa6ef2751cae773b9, State: Running, Role: LEADER
I20260925 19:00:19.142647 5475 consensus_queue.cc:237] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 } } peers { permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 } } peers { permanent_uuid: "84019a09f0374b9aa6ef2751cae773b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32959 } }
I20260925 19:00:19.143481 5004 catalog_manager.cc:5495] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84019a09f0374b9aa6ef2751cae773b9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "84019a09f0374b9aa6ef2751cae773b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84019a09f0374b9aa6ef2751cae773b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32959 } health_report { overall_health: HEALTHY } } }
I20260925 19:00:19.147789 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.146895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.0.98.2:32959)
0925 19:00:19.146954 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.146956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.146964 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:19.146996 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.146997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.147018 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.147104 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.147580 (+ 476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.147581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.147582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.147586 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.147599 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.147603 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.147660 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.147713 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.147726 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:00:19.149286 5408 raft_consensus.cc:1261] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c [term 1 FOLLOWER]: Refusing update from remote peer 84019a09f0374b9aa6ef2751cae773b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:19.149302 5146 raft_consensus.cc:1261] T 5e2248cd04174c98b1b2a65c9be34fc0 P faa6a63220ac4b1eaed2770ec1e6c1fc [term 1 FOLLOWER]: Refusing update from remote peer 84019a09f0374b9aa6ef2751cae773b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:19.149631 5475 consensus_queue.cc:1035] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa6a63220ac4b1eaed2770ec1e6c1fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:19.149758 5478 consensus_queue.cc:1035] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a05166bf704d85ac164b9c1693e59c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:19.154215 5003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:19.154330 5003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:19.154983 5003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f59cafaee6842ff9c7fca5b912dbcc1: Sending DeleteTablet for 3 replicas of tablet 5e2248cd04174c98b1b2a65c9be34fc0
I20260925 19:00:19.155259 5257 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2248cd04174c98b1b2a65c9be34fc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:19 UTC) from {username='slave'} at 127.0.0.1:37818
I20260925 19:00:19.155274 5126 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2248cd04174c98b1b2a65c9be34fc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:19 UTC) from {username='slave'} at 127.0.0.1:33634
I20260925 19:00:19.155539 5491 tablet_replica.cc:330] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: stopping tablet replica
I20260925 19:00:19.155550 5388 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2248cd04174c98b1b2a65c9be34fc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:19 UTC) from {username='slave'} at 127.0.0.1:52568
I20260925 19:00:19.155630 5491 raft_consensus.cc:2229] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:19.155721 5491 raft_consensus.cc:2258] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:19.155757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 5064
I20260925 19:00:19.155848 5493 tablet_replica.cc:330] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: stopping tablet replica
I20260925 19:00:19.155965 5493 raft_consensus.cc:2229] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:19.156090 5493 raft_consensus.cc:2258] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:19.156118 5491 ts_tablet_manager.cc:1905] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:19.156529 5493 ts_tablet_manager.cc:1905] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:19.157655 5491 ts_tablet_manager.cc:1918] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:19.157727 5491 log.cc:1196] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/5e2248cd04174c98b1b2a65c9be34fc0
I20260925 19:00:19.158007 5491 ts_tablet_manager.cc:1939] T 5e2248cd04174c98b1b2a65c9be34fc0 P 84019a09f0374b9aa6ef2751cae773b9: Deleting consensus metadata
I20260925 19:00:19.158466 4990 catalog_manager.cc:4841] TS 84019a09f0374b9aa6ef2751cae773b9 (127.0.98.2:32959): tablet 5e2248cd04174c98b1b2a65c9be34fc0 (table test-table [id=adac823aa06d44ef8509b23e2b305574]) successfully deleted
I20260925 19:00:19.158676 5493 ts_tablet_manager.cc:1918] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:19.158746 5493 log.cc:1196] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/5e2248cd04174c98b1b2a65c9be34fc0
I20260925 19:00:19.158973 5493 ts_tablet_manager.cc:1939] T 5e2248cd04174c98b1b2a65c9be34fc0 P 44a05166bf704d85ac164b9c1693e59c: Deleting consensus metadata
I20260925 19:00:19.159487 4991 catalog_manager.cc:4841] TS 44a05166bf704d85ac164b9c1693e59c (127.0.98.3:41191): tablet 5e2248cd04174c98b1b2a65c9be34fc0 (table test-table [id=adac823aa06d44ef8509b23e2b305574]) successfully deleted
W20260925 19:00:19.162357 4988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40727 (error 108)
I20260925 19:00:19.162590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 5195
I20260925 19:00:19.163146 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.162676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40727 (local address 127.0.0.1:33640)
0925 19:00:19.163074 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:19.163111 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379}
W20260925 19:00:19.163246 4988 catalog_manager.cc:4584] TS faa6a63220ac4b1eaed2770ec1e6c1fc (127.0.98.1:40727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e2248cd04174c98b1b2a65c9be34fc0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40727: connect: Connection refused (error 111)
I20260925 19:00:19.166630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 5326
I20260925 19:00:19.171109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 4973
2026-09-25T19:00:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (600 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 19:00:19.192276 392 mini_oidc.cc:152] Starting JWKS server
I20260925 19:00:19.193279 392 webserver.cc:460] Webserver started at https://127.0.0.1:44631/ using document root <none> and password file <none>
I20260925 19:00:19.193414 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 19:00:19.193593 392 webserver.cc:460] Webserver started at http://127.0.0.1:36875/ using document root <none> and password file <none>
2026-09-25T19:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:19Z Disabled control of system clock
I20260925 19:00:19.201282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41813
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41813
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44631/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/testchainca.cert with env {}
W20260925 19:00:19.292529 5500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:19.292707 5500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:19.292728 5500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:19.293906 5500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:19.293985 5500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:19.294008 5500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:19.294032 5500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:19.294056 5500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:00:19.294083 5500 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:00:19.294102 5500 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44631/jwks.json
I20260925 19:00:19.295568 5500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44631/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:19.295739 5500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:19.295904 5500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:19.300568 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:19.300580 5500 server_base.cc:1034] running on GCE node
W20260925 19:00:19.300688 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:19.300575 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:19.300983 5500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:19.301293 5500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:19.302448 5500 hybrid_clock.cc:648] HybridClock initialized: now 1790362819302444 us; error 35 us; skew 500 ppm
I20260925 19:00:19.303587 5500 webserver.cc:460] Webserver started at http://127.0.98.62:32949/ using document root <none> and password file <none>
I20260925 19:00:19.303808 5500 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:19.303876 5500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:19.304028 5500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:19.304987 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "c9f2f70c358140879d4ef85f53a236a3"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "7424c706f4210984786c3f2c969bae41"
server_key_iv: "a1ee92b8c06b639c33790d2626771bbf"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.305331 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9f2f70c358140879d4ef85f53a236a3"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "7424c706f4210984786c3f2c969bae41"
server_key_iv: "a1ee92b8c06b639c33790d2626771bbf"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.306536 5500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:19.307194 5514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:19.307350 5500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:19.307448 5500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "c9f2f70c358140879d4ef85f53a236a3"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "7424c706f4210984786c3f2c969bae41"
server_key_iv: "a1ee92b8c06b639c33790d2626771bbf"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.307518 5500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:19.327477 5500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:19.327752 5500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:19.327890 5500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:19.331580 5500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41813
I20260925 19:00:19.331610 5566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41813 every 8 connection(s)
I20260925 19:00:19.331918 5500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:19.332876 5567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:19.335062 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3: Bootstrap starting.
I20260925 19:00:19.335647 5567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:19.335929 5567 log.cc:824] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:19.336728 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3: No bootstrap required, opened a new log
I20260925 19:00:19.338203 5567 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f2f70c358140879d4ef85f53a236a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } }
I20260925 19:00:19.338356 5567 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:19.338414 5567 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f2f70c358140879d4ef85f53a236a3, State: Initialized, Role: FOLLOWER
I20260925 19:00:19.338572 5567 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f2f70c358140879d4ef85f53a236a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } }
I20260925 19:00:19.338653 5567 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:19.338702 5567 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:19.338750 5567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:19.339244 5567 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f2f70c358140879d4ef85f53a236a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } }
I20260925 19:00:19.339357 5567 leader_election.cc:302] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9f2f70c358140879d4ef85f53a236a3; no voters:
I20260925 19:00:19.339535 5567 leader_election.cc:288] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:19.339617 5570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:19.339715 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 5500
I20260925 19:00:19.339782 5567 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:19.339813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:19.339818 5570 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [term 1 LEADER]: Becoming Leader. State: Replica: c9f2f70c358140879d4ef85f53a236a3, State: Running, Role: LEADER
I20260925 19:00:19.339923 5570 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f2f70c358140879d4ef85f53a236a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } }
I20260925 19:00:19.339993 392 external_mini_cluster.cc:1428] Setting key 5e0eed2cde0b23ae52461506bcb1846b
I20260925 19:00:19.340332 5572 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9f2f70c358140879d4ef85f53a236a3. Latest consensus state: current_term: 1 leader_uuid: "c9f2f70c358140879d4ef85f53a236a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f2f70c358140879d4ef85f53a236a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } } }
I20260925 19:00:19.340408 5572 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:19.340600 5571 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9f2f70c358140879d4ef85f53a236a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f2f70c358140879d4ef85f53a236a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } } }
I20260925 19:00:19.340631 5580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:19.340739 5571 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:19.341159 5580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:19.342909 5580 catalog_manager.cc:1347] Generated new cluster ID: 1608a6bff9c84f8eb9cc70f4e93e7d5a
I20260925 19:00:19.342958 5580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:19.346395 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.340426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.0.98.62:41813)
0925 19:00:19.340648 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.340656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.341194 (+ 538us) server_negotiation.cc:407] Connection header received
0925 19:00:19.341344 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.341370 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.341433 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.341559 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.341919 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.343211 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.344298 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.344891 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.344999 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:19.345681 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.345685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.345685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.345895 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.345930 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.345943 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.346021 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.346113 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.346181 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260925 19:00:19.347219 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:00:19.352679 5580 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:19.353338 5580 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:19.355834 5580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9f2f70c358140879d4ef85f53a236a3: Generated new TSK 0
I20260925 19:00:19.355983 5580 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 19:00:19.385462 5597 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 19:00:19.385550 5597 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260925 19:00:19.385967 5596 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:00:19.363301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41813 (local address 127.0.0.1:51288)
0925 19:00:19.363471 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:19.363481 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:19.363546 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:19.363696 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:19.363697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:19.363699 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:00:19.363757 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:19.363758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.364322 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.364322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:19.365267 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:19.365270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.365624 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.365627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:19.365693 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:19.365714 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:00:19.385658 (+ 19944us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44631/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 19:00:19.385748 (+ 90us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44631/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44631/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 19:00:19.386385 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.363385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.0.98.62:41813)
0925 19:00:19.363424 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.363426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.363543 (+ 117us) server_negotiation.cc:407] Connection header received
0925 19:00:19.363625 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.363626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.363635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.363684 (+ 49us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:00:19.363813 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.364225 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.365367 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.365541 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.365606 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:19.365822 (+ 216us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:00:19.385474 (+ 19652us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44631/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 19:00:19.386190 (+ 716us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44631/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:00:19.386402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 5500
2026-09-25T19:00:19Z chronyd exiting
I20260925 19:00:19.421515 392 test_util.cc:175] -----------------------------------------------
I20260925 19:00:19.421578 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790362794046639-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 19:00:19.422389 392 mini_oidc.cc:152] Starting JWKS server
I20260925 19:00:19.423046 392 webserver.cc:460] Webserver started at https://127.0.0.1:46121/ using document root <none> and password file <none>
I20260925 19:00:19.423089 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 19:00:19.423259 392 webserver.cc:460] Webserver started at http://127.0.0.1:38547/ using document root <none> and password file <none>
2026-09-25T19:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:19Z Disabled control of system clock
I20260925 19:00:19.430472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40109
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40109
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46121/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/testchainca.cert with env {}
W20260925 19:00:19.505371 5603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:19.505548 5603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:19.505568 5603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:19.506740 5603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:19.506779 5603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:19.506790 5603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:19.506821 5603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:19.506845 5603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:00:19.506870 5603 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:00:19.506894 5603 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46121/jwks.json
I20260925 19:00:19.508301 5603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46121/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:19.508528 5603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:19.508723 5603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:00:19.513321 5603 server_base.cc:1034] running on GCE node
W20260925 19:00:19.513309 5608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:19.513284 5609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:19.513293 5611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:19.513945 5603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:19.514230 5603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:19.515414 5603 hybrid_clock.cc:648] HybridClock initialized: now 1790362819515397 us; error 39 us; skew 500 ppm
I20260925 19:00:19.516495 5603 webserver.cc:460] Webserver started at http://127.0.98.62:33241/ using document root <none> and password file <none>
I20260925 19:00:19.516727 5603 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:19.516793 5603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:19.516921 5603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:19.517874 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "81197d406cc94fcf94519a138dd0c145"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "11afc2594e898ff087b6429f19475c35"
server_key_iv: "a0283ea6fd6fea8f932e7909ac4757a7"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.518216 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "81197d406cc94fcf94519a138dd0c145"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "11afc2594e898ff087b6429f19475c35"
server_key_iv: "a0283ea6fd6fea8f932e7909ac4757a7"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.519449 5603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:00:19.520130 5617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:19.520305 5603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:19.520403 5603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "81197d406cc94fcf94519a138dd0c145"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "11afc2594e898ff087b6429f19475c35"
server_key_iv: "a0283ea6fd6fea8f932e7909ac4757a7"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.520500 5603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:19.539438 5603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:19.539713 5603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:19.539863 5603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:19.543624 5603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40109
I20260925 19:00:19.543656 5669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40109 every 8 connection(s)
I20260925 19:00:19.543964 5603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:19.544817 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:19.546813 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145: Bootstrap starting.
I20260925 19:00:19.547331 5670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:19.547595 5670 log.cc:824] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:19.548211 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145: No bootstrap required, opened a new log
I20260925 19:00:19.549666 5670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81197d406cc94fcf94519a138dd0c145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40109 } }
I20260925 19:00:19.549793 5670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:19.549815 5670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81197d406cc94fcf94519a138dd0c145, State: Initialized, Role: FOLLOWER
I20260925 19:00:19.549996 5670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81197d406cc94fcf94519a138dd0c145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40109 } }
I20260925 19:00:19.550076 5670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:19.550118 5670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:19.550166 5670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:19.550676 5670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81197d406cc94fcf94519a138dd0c145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40109 } }
I20260925 19:00:19.550804 5670 leader_election.cc:302] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81197d406cc94fcf94519a138dd0c145; no voters:
I20260925 19:00:19.550969 5670 leader_election.cc:288] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:19.550997 5673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:19.551170 5673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [term 1 LEADER]: Becoming Leader. State: Replica: 81197d406cc94fcf94519a138dd0c145, State: Running, Role: LEADER
I20260925 19:00:19.551194 5670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:19.551294 5673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81197d406cc94fcf94519a138dd0c145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40109 } }
I20260925 19:00:19.551704 5674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81197d406cc94fcf94519a138dd0c145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81197d406cc94fcf94519a138dd0c145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40109 } } }
I20260925 19:00:19.551816 5675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81197d406cc94fcf94519a138dd0c145. Latest consensus state: current_term: 1 leader_uuid: "81197d406cc94fcf94519a138dd0c145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81197d406cc94fcf94519a138dd0c145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40109 } } }
I20260925 19:00:19.551898 5675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:19.551808 5674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:19.552134 5682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:19.552685 5682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:19.554088 5682 catalog_manager.cc:1347] Generated new cluster ID: 006bed4b2b454dcfbf3c7bd6fb88341e
I20260925 19:00:19.554140 5682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:19.554207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 5603
I20260925 19:00:19.554311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:19.554487 392 external_mini_cluster.cc:1428] Setting key 3b85e87364a3a5daad9c68b5336d761f
I20260925 19:00:19.562860 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.554855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.0.98.62:40109)
0925 19:00:19.554997 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.555010 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.558442 (+ 3432us) server_negotiation.cc:407] Connection header received
0925 19:00:19.558621 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.558626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.558674 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.558804 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.559002 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.560166 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.561008 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.561411 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.561535 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:19.562172 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.562175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.562176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.562392 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.562424 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.562434 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.562499 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.562589 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.562667 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":60,"threads_started":1}
I20260925 19:00:19.563639 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:00:19.587872 5682 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:19.588522 5682 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:19.590974 5682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81197d406cc94fcf94519a138dd0c145: Generated new TSK 0
I20260925 19:00:19.591140 5682 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 19:00:19.613068 5700 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 19:00:19.613142 5700 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 19:00:19.613440 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.599999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.0.98.62:40109)
0925 19:00:19.600042 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.600044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.600096 (+ 52us) server_negotiation.cc:407] Connection header received
0925 19:00:19.600156 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.600157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.600169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.600248 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:00:19.600347 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.601093 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.601807 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.602108 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.602177 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:19.602333 (+ 156us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:00:19.613243 (+ 10910us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 19:00:19.613367 (+ 124us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39734: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 19:00:19.613487 5693 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39734: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 19:00:19.613502 5699 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:00:19.599897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40109 (local address 127.0.0.1:39734)
0925 19:00:19.600047 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:19.600057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:19.600108 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:19.600252 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:19.600252 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:19.600254 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:00:19.600298 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:19.600299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.601177 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.601177 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:19.601726 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:19.601727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.602179 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.602182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:19.602218 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:19.602222 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:00:19.613333 (+ 11111us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 19:00:19.613407 (+ 74us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40109: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260925 19:00:19.613772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 5603
2026-09-25T19:00:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 19:00:19.650695 392 mini_oidc.cc:152] Starting JWKS server
I20260925 19:00:19.651333 392 webserver.cc:460] Webserver started at https://127.0.0.1:32825/ using document root <none> and password file <none>
I20260925 19:00:19.651371 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 19:00:19.651525 392 webserver.cc:460] Webserver started at http://127.0.0.1:43245/ using document root <none> and password file <none>
2026-09-25T19:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:19Z Disabled control of system clock
I20260925 19:00:19.658378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44373
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44373
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32825/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 19:00:19.732860 5706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:19.733040 5706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:19.733060 5706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:19.734052 5706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:19.734083 5706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:19.734095 5706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:19.734107 5706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:19.734117 5706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:00:19.734128 5706 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:00:19.734143 5706 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32825/jwks.json
I20260925 19:00:19.735384 5706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32825/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:19.735553 5706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:19.735749 5706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:00:19.740257 5706 server_base.cc:1034] running on GCE node
W20260925 19:00:19.740281 5711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:19.740388 5714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:19.740281 5712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:19.740736 5706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:19.741008 5706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:19.742166 5706 hybrid_clock.cc:648] HybridClock initialized: now 1790362819742155 us; error 54 us; skew 500 ppm
I20260925 19:00:19.743281 5706 webserver.cc:460] Webserver started at http://127.0.98.62:34879/ using document root <none> and password file <none>
I20260925 19:00:19.743505 5706 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:19.743568 5706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:19.743716 5706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:19.744706 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "4cf61186615d4ba98c990064b3272e0d"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "0946a1dedd00ac030ea9fb12d9daf6b9"
server_key_iv: "08e778d3e4b4e1948c692e7eae869df3"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.745057 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "4cf61186615d4ba98c990064b3272e0d"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "0946a1dedd00ac030ea9fb12d9daf6b9"
server_key_iv: "08e778d3e4b4e1948c692e7eae869df3"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.746297 5706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:00:19.746969 5720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:19.747112 5706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:19.747213 5706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "4cf61186615d4ba98c990064b3272e0d"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "0946a1dedd00ac030ea9fb12d9daf6b9"
server_key_iv: "08e778d3e4b4e1948c692e7eae869df3"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.747295 5706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:19.759554 5706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:19.759812 5706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:19.759951 5706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:19.763422 5706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44373
I20260925 19:00:19.763450 5772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44373 every 8 connection(s)
I20260925 19:00:19.763734 5706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:19.764310 5773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:19.766579 5773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d: Bootstrap starting.
I20260925 19:00:19.767046 5773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:19.767274 5773 log.cc:824] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:19.767827 5773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d: No bootstrap required, opened a new log
I20260925 19:00:19.769140 5773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf61186615d4ba98c990064b3272e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } }
I20260925 19:00:19.769261 5773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:19.769281 5773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf61186615d4ba98c990064b3272e0d, State: Initialized, Role: FOLLOWER
I20260925 19:00:19.769423 5773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf61186615d4ba98c990064b3272e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } }
I20260925 19:00:19.769500 5773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:19.769523 5773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:19.769548 5773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:19.769991 5773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf61186615d4ba98c990064b3272e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } }
I20260925 19:00:19.770069 5773 leader_election.cc:302] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cf61186615d4ba98c990064b3272e0d; no voters:
I20260925 19:00:19.770182 5773 leader_election.cc:288] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:19.770274 5776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:19.770447 5776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [term 1 LEADER]: Becoming Leader. State: Replica: 4cf61186615d4ba98c990064b3272e0d, State: Running, Role: LEADER
I20260925 19:00:19.770457 5773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:19.770551 5776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf61186615d4ba98c990064b3272e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } }
I20260925 19:00:19.770921 5777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cf61186615d4ba98c990064b3272e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf61186615d4ba98c990064b3272e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } } }
I20260925 19:00:19.771010 5777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:19.771021 5778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cf61186615d4ba98c990064b3272e0d. Latest consensus state: current_term: 1 leader_uuid: "4cf61186615d4ba98c990064b3272e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf61186615d4ba98c990064b3272e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } } }
I20260925 19:00:19.771080 5778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:19.771234 5784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:19.771847 5784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:19.772111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 5706
I20260925 19:00:19.772202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:19.772367 392 external_mini_cluster.cc:1428] Setting key 236c8bf4f72a86292483d138f3f0dc93
I20260925 19:00:19.773380 5784 catalog_manager.cc:1347] Generated new cluster ID: 575c69d548c243ddb812516edbbf3716
I20260925 19:00:19.773427 5784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:19.777062 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.772908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.0.98.62:44373)
0925 19:00:19.773083 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.773090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.773120 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:00:19.773197 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.773202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.773248 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.773422 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.773597 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.774492 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.775221 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.775630 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.775770 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:19.776440 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.776444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.776444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.776609 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.776641 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.776651 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.776726 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.776798 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.776877 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":98,"threads_started":1}
I20260925 19:00:19.778203 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:00:19.794958 5784 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:19.795576 5784 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:19.797461 5784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cf61186615d4ba98c990064b3272e0d: Generated new TSK 0
I20260925 19:00:19.797626 5784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:00:19.811335 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.807971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.0.98.62:44373)
0925 19:00:19.808012 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.808014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.808102 (+ 88us) server_negotiation.cc:407] Connection header received
0925 19:00:19.808165 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.808165 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.808174 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.808224 (+ 50us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:00:19.808335 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.808772 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.809893 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.810061 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.810108 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:19.810330 (+ 222us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:00:19.811154 (+ 824us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 19:00:19.811275 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45054: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 19:00:19.811379 5795 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45054: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 19:00:19.811437 5802 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:00:19.807865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44373 (local address 127.0.0.1:45054)
0925 19:00:19.808014 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:19.808025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:19.808099 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:19.808233 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:19.808234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:19.808236 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:00:19.808282 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:19.808284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.808843 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.808844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:19.809811 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:19.809813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.810134 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.810139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:19.810205 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:19.810210 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:00:19.811250 (+ 1040us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 19:00:19.811343 (+ 93us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44373: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260925 19:00:19.811626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 5706
E20260925 19:00:19.822006 5803 webserver.cc:533] Webserver: sslize failed:
E20260925 19:00:19.822062 5803 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T19:00:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (197 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T19:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:19Z Disabled control of system clock
I20260925 19:00:19.855803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45987
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45987
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/testchainca.cert with env {}
W20260925 19:00:19.932379 5807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:19.932561 5807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:19.932582 5807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:19.933670 5807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:19.933708 5807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:19.933722 5807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:19.933769 5807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:19.933794 5807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 19:00:19.933820 5807 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 19:00:19.933868 5807 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 19:00:19.935384 5807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:19.935616 5807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:19.935804 5807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:19.940163 5815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:19.940187 5812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:19.940253 5813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:19.940461 5807 server_base.cc:1034] running on GCE node
I20260925 19:00:19.940722 5807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:19.940964 5807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:19.942121 5807 hybrid_clock.cc:648] HybridClock initialized: now 1790362819942113 us; error 36 us; skew 500 ppm
I20260925 19:00:19.943243 5807 webserver.cc:460] Webserver started at http://127.0.98.62:34083/ using document root <none> and password file <none>
I20260925 19:00:19.943477 5807 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:19.943547 5807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:19.943673 5807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:19.944618 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "d7ceb614ae5545c0ab24b9907c98028e"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "edfbc398c96a25129f6349c4efb6977b"
server_key_iv: "c2e9669e5d3116d75000636ac593999b"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.944955 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7ceb614ae5545c0ab24b9907c98028e"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "edfbc398c96a25129f6349c4efb6977b"
server_key_iv: "c2e9669e5d3116d75000636ac593999b"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.946076 5807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:00:19.946780 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:19.946942 5807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:19.947031 5807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "d7ceb614ae5545c0ab24b9907c98028e"
format_stamp: "Formatted at 2026-09-25 19:00:19 on dist-test-slave-49bz"
server_key: "edfbc398c96a25129f6349c4efb6977b"
server_key_iv: "c2e9669e5d3116d75000636ac593999b"
server_key_version: "encryptionkey@0"
I20260925 19:00:19.947108 5807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:19.962174 5807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:19.962472 5807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:19.962618 5807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:19.966066 5807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45987
I20260925 19:00:19.966102 5873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45987 every 8 connection(s)
I20260925 19:00:19.966470 5807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:19.967340 5874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:19.969434 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e: Bootstrap starting.
I20260925 19:00:19.969470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 5807
I20260925 19:00:19.969563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:19.969743 392 external_mini_cluster.cc:1428] Setting key c7d1e9b2e3400f38b54963eec59cbd51
I20260925 19:00:19.970108 5874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:19.970454 5874 log.cc:824] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:19.971045 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e: No bootstrap required, opened a new log
I20260925 19:00:19.972831 5874 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ceb614ae5545c0ab24b9907c98028e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45987 } }
I20260925 19:00:19.972971 5874 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:19.973019 5874 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ceb614ae5545c0ab24b9907c98028e, State: Initialized, Role: FOLLOWER
I20260925 19:00:19.973148 5874 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ceb614ae5545c0ab24b9907c98028e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45987 } }
I20260925 19:00:19.973227 5874 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:19.973268 5874 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:19.973315 5874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:19.973817 5874 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ceb614ae5545c0ab24b9907c98028e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45987 } }
I20260925 19:00:19.974164 5874 leader_election.cc:302] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7ceb614ae5545c0ab24b9907c98028e; no voters:
I20260925 19:00:19.974359 5874 leader_election.cc:288] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:19.974380 5879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:19.974501 5879 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [term 1 LEADER]: Becoming Leader. State: Replica: d7ceb614ae5545c0ab24b9907c98028e, State: Running, Role: LEADER
I20260925 19:00:19.974576 5879 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ceb614ae5545c0ab24b9907c98028e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45987 } }
I20260925 19:00:19.974645 5874 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:19.974881 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ceb614ae5545c0ab24b9907c98028e. Latest consensus state: current_term: 1 leader_uuid: "d7ceb614ae5545c0ab24b9907c98028e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ceb614ae5545c0ab24b9907c98028e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45987 } } }
I20260925 19:00:19.974963 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:19.975104 5880 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7ceb614ae5545c0ab24b9907c98028e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ceb614ae5545c0ab24b9907c98028e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45987 } } }
I20260925 19:00:19.975160 5880 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:19.975180 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:19.970194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.0.98.62:45987)
0925 19:00:19.970425 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:19.970434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:19.970464 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:00:19.970917 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:19.970923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:19.970974 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:19.971109 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:19.971346 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.972355 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.973103 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:19.973606 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:19.973738 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:19.974296 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:19.974300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:19.974300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:00:19.974528 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:19.974567 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:00:19.974584 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:19.974871 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:19.974901 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 19:00:19.974976 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260925 19:00:19.975281 5884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:19.976116 5884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:19.977065 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:00:19.977833 5884 catalog_manager.cc:1347] Generated new cluster ID: b669b92a5fef4c9fa74e13d9c004d730
I20260925 19:00:19.977876 5884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:19.982144 5884 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:19.982834 5884 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:19.989050 5884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7ceb614ae5545c0ab24b9907c98028e: Generated new TSK 0
I20260925 19:00:19.989218 5884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:00:20.058297 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:20.000512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48460 (local address 127.0.98.62:45987)
0925 19:00:20.000582 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:20.000583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:20.000688 (+ 105us) server_negotiation.cc:407] Connection header received
0925 19:00:20.000750 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:20.000751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:20.000759 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:20.000826 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0925 19:00:20.000980 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:20.001444 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:20.002541 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:20.002718 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:20.002776 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:20.002942 (+ 166us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 19:00:20.058074 (+ 55132us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 19:00:20.058200 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48460: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 19:00:20.058382 5877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48460: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 19:00:20.058466 5903 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:00:20.000460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45987 (local address 127.0.0.1:48460)
0925 19:00:20.000618 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:20.000629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:20.000693 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:20.000846 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:20.000847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:20.000848 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 19:00:20.000920 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:20.000922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:20.001514 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:20.001515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:20.002467 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:20.002469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:20.002794 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:20.002800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:20.002857 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:20.002862 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 19:00:20.058212 (+ 55350us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 19:00:20.058368 (+ 156us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45987: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":112,"threads_started":1}
I20260925 19:00:20.058815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 5807
2026-09-25T19:00:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (226 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (83 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (82 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:20 dist-test-slave-49bz krb5kdc[5915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:20 dist-test-slave-49bz krb5kdc[5915](info): set up 2 sockets
Sep 25 19:00:20 dist-test-slave-49bz krb5kdc[5915](info): commencing operation
krb5kdc: starting...
W20260925 19:00:21.210448 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:00:21.318619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38331
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:21.393216 5928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:21.393391 5928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:21.393410 5928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:21.394728 5928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:21.394778 5928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:21.394807 5928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:21.394819 5928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:21.394877 5928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:21.396458 5928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:21.396693 5928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:21.396910 5928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:21.401484 5933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:21.401515 5936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:21.401515 5934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:21.401710 5928 server_base.cc:1034] running on GCE node
I20260925 19:00:21.401938 5928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:21.402215 5928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:21.403398 5928 hybrid_clock.cc:648] HybridClock initialized: now 1790362821403379 us; error 34 us; skew 500 ppm
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.412108 5928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:00:21.412521 5928 webserver.cc:460] Webserver started at http://127.0.98.62:33171/ using document root <none> and password file <none>
I20260925 19:00:21.412750 5928 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:21.412817 5928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:21.412961 5928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:21.413962 5928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "ee0e892be2f846ffa3421af68f1055cf"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "1fce8841b5872ba5075a6e85cdfcc68e"
server_key_iv: "76f905210617fe3fe4c75d53ee60d52f"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.414372 5928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee0e892be2f846ffa3421af68f1055cf"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "1fce8841b5872ba5075a6e85cdfcc68e"
server_key_iv: "76f905210617fe3fe4c75d53ee60d52f"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.415688 5928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:21.416493 5943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.416688 5928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:21.416785 5928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "ee0e892be2f846ffa3421af68f1055cf"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "1fce8841b5872ba5075a6e85cdfcc68e"
server_key_iv: "76f905210617fe3fe4c75d53ee60d52f"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.416872 5928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:21.434722 5928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:21.436877 5928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:21.437026 5928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:21.440725 5928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38331
I20260925 19:00:21.440789 5995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38331 every 8 connection(s)
I20260925 19:00:21.441224 5928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:21.441862 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:21.442448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 5928
I20260925 19:00:21.442562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:21.442734 392 external_mini_cluster.cc:1428] Setting key 35e4a26b9fad018f2d7044afe7d6eca4
I20260925 19:00:21.444638 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf: Bootstrap starting.
I20260925 19:00:21.445323 5996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:21.445602 5996 log.cc:824] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:21.446242 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf: No bootstrap required, opened a new log
I20260925 19:00:21.448307 5996 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e892be2f846ffa3421af68f1055cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38331 } }
I20260925 19:00:21.448450 5996 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:21.448493 5996 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0e892be2f846ffa3421af68f1055cf, State: Initialized, Role: FOLLOWER
I20260925 19:00:21.448618 5996 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e892be2f846ffa3421af68f1055cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38331 } }
I20260925 19:00:21.448696 5996 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:21.448730 5996 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:21.448777 5996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [term 0 FOLLOWER]: Advancing to term 1
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:21.449241 5996 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e892be2f846ffa3421af68f1055cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38331 } }
I20260925 19:00:21.449363 5996 leader_election.cc:302] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee0e892be2f846ffa3421af68f1055cf; no voters:
I20260925 19:00:21.449512 5996 leader_election.cc:288] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:21.449586 6001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:21.449740 6001 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [term 1 LEADER]: Becoming Leader. State: Replica: ee0e892be2f846ffa3421af68f1055cf, State: Running, Role: LEADER
I20260925 19:00:21.449791 5996 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:21.449841 6001 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e892be2f846ffa3421af68f1055cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38331 } }
I20260925 19:00:21.450167 6001 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee0e892be2f846ffa3421af68f1055cf. Latest consensus state: current_term: 1 leader_uuid: "ee0e892be2f846ffa3421af68f1055cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e892be2f846ffa3421af68f1055cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38331 } } }
I20260925 19:00:21.450239 6001 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:21.450469 6008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:21.450490 6002 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee0e892be2f846ffa3421af68f1055cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0e892be2f846ffa3421af68f1055cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38331 } } }
I20260925 19:00:21.450651 6002 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:21.450968 6008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:21.453099 6008 catalog_manager.cc:1347] Generated new cluster ID: 0067c85fa8104064995505ff970954fe
I20260925 19:00:21.453146 6008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:21.453639 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:21.443193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.0.98.62:38331)
0925 19:00:21.443361 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:21.443370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:21.443404 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:00:21.443946 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:21.443954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:21.444013 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:21.444156 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:21.444715 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.445996 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.446808 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.447399 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.447506 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:21.449799 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:21.449803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:21.449804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:21.449836 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:21.451803 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.452207 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.452209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.452210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.452324 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.452542 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.452542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.452543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.452835 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:21.452970 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:21.453322 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 19:00:21.453453 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260925 19:00:21.472018 6008 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:21.472488 6008 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:21.485710 6008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee0e892be2f846ffa3421af68f1055cf: Generated new TSK 0
I20260925 19:00:21.485879 6008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:00:21.515048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38331
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:21.591411 6022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:21.591593 6022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:21.591615 6022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:21.592636 6022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:21.592669 6022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:21.592694 6022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:21.593971 6022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:21.594128 6022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:21.594302 6022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:21.598943 6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:21.598997 6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:21.599105 6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:21.599005 6022 server_base.cc:1034] running on GCE node
I20260925 19:00:21.599360 6022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:21.599612 6022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:21.600790 6022 hybrid_clock.cc:648] HybridClock initialized: now 1790362821600767 us; error 37 us; skew 500 ppm
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.603204 6022 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:00:21.603642 6022 webserver.cc:460] Webserver started at http://127.0.98.1:46437/ using document root <none> and password file <none>
I20260925 19:00:21.603860 6022 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:21.603904 6022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:21.604041 6022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:21.605033 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "ebe644690748425285700bed22a97ed8"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "f474017239daf246b000a5933adf7a02"
server_key_iv: "4c73dd7a9891e9795593bf4fa1acf4a9"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.605367 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ebe644690748425285700bed22a97ed8"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "f474017239daf246b000a5933adf7a02"
server_key_iv: "4c73dd7a9891e9795593bf4fa1acf4a9"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.606614 6022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:00:21.607296 6037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.607481 6022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:21.607566 6022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "ebe644690748425285700bed22a97ed8"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "f474017239daf246b000a5933adf7a02"
server_key_iv: "4c73dd7a9891e9795593bf4fa1acf4a9"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.607635 6022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:21.616695 6022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:21.618402 6022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:21.618546 6022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:21.618747 6022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:21.619047 6022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:21.619102 6022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.619135 6022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:21.619163 6022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.624914 6022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35761
I20260925 19:00:21.624966 6150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35761 every 8 connection(s)
I20260925 19:00:21.625234 6022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:00:21.629048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 6022
I20260925 19:00:21.629149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:21.629315 392 external_mini_cluster.cc:1428] Setting key de5e2b5813f0d86c9a2a8fb910f55028
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:21.635094 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:21.625853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38331 (local address 127.0.98.1:35031)
0925 19:00:21.626005 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:21.626041 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:21.626981 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:21.627187 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:21.627189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:21.627422 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:21.627594 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:21.627598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.628242 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.628243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:21.628942 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:21.628944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.629442 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.629443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:21.629599 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:21.630131 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:21.630141 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:21.631916 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:21.633768 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:21.633791 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:21.633795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:21.634038 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:21.634292 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:21.634293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:21.634293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:21.634391 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:21.634673 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:21.634676 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:21.634780 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 19:00:21.634849 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:00:21.634953 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260925 19:00:21.635138 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:21.625891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35031 (local address 127.0.98.62:38331)
0925 19:00:21.625951 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:21.625953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:21.626118 (+ 165us) server_negotiation.cc:407] Connection header received
0925 19:00:21.627067 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:21.627067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:21.627077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:21.627127 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:21.627661 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.628147 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.629043 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.629327 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.629404 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:21.632008 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:21.632010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:21.632010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:21.632017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:21.633686 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.634113 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.634115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.634115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.634177 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.634458 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.634459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.634460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.634602 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:21.634682 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:21.634915 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 19:00:21.635003 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:00:21.635676 6151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38331
I20260925 19:00:21.635785 6151 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:21.635948 6151 heartbeater.cc:511] Master 127.0.98.62:38331 requested a full tablet report, sending...
I20260925 19:00:21.636431 5960 ts_manager.cc:194] Registered new tserver with Master: ebe644690748425285700bed22a97ed8 (127.0.98.1:35761)
I20260925 19:00:21.637382 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35031
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:00:21.654033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38331
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:21.728884 6156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:21.729055 6156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:21.729079 6156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:21.730201 6156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:21.730242 6156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:21.730338 6156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:21.731739 6156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:21.731978 6156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:21.732177 6156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:21.736677 6164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:21.736685 6162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:21.736743 6161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:21.736855 6156 server_base.cc:1034] running on GCE node
I20260925 19:00:21.737105 6156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:21.737357 6156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:21.738512 6156 hybrid_clock.cc:648] HybridClock initialized: now 1790362821738504 us; error 37 us; skew 500 ppm
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.740878 6156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:00:21.741299 6156 webserver.cc:460] Webserver started at http://127.0.98.2:33335/ using document root <none> and password file <none>
I20260925 19:00:21.741508 6156 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:21.741546 6156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:21.741673 6156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:21.742746 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "ab4aa8afcf4f40ecb5713cf104c0f346"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "5db9e82f2b7a1c630e6187d713710300"
server_key_iv: "dc1847d9b54671e15f070e20ebd81c7a"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.743083 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ab4aa8afcf4f40ecb5713cf104c0f346"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "5db9e82f2b7a1c630e6187d713710300"
server_key_iv: "dc1847d9b54671e15f070e20ebd81c7a"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.744326 6156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:00:21.744979 6171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.745141 6156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:21.745213 6156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "ab4aa8afcf4f40ecb5713cf104c0f346"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "5db9e82f2b7a1c630e6187d713710300"
server_key_iv: "dc1847d9b54671e15f070e20ebd81c7a"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.745260 6156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:21.756075 6156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:21.757715 6156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:21.757817 6156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:21.757972 6156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:21.758194 6156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:21.758221 6156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.758241 6156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:21.758320 6156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.763686 6156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33343
I20260925 19:00:21.763760 6284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33343 every 8 connection(s)
I20260925 19:00:21.764034 6156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:00:21.767810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 6156
I20260925 19:00:21.767920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:00:21.768077 392 external_mini_cluster.cc:1428] Setting key 7793c20501503649244badfd395b292a
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:21.772838 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:21.764686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36299 (local address 127.0.98.62:38331)
0925 19:00:21.764735 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:21.764737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:21.764948 (+ 211us) server_negotiation.cc:407] Connection header received
0925 19:00:21.765742 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:21.765743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:21.765752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:21.765810 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:21.766341 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.766796 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.767667 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.767841 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.767917 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:21.770221 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:21.770223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:21.770223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:21.770230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:21.771589 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.771972 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.771973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.771973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.772028 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.772251 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.772252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.772253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.772360 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:21.772430 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:21.772674 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 19:00:21.772748 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:00:21.772970 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:21.764668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38331 (local address 127.0.98.2:36299)
0925 19:00:21.764834 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:21.764864 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:21.765661 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:21.765862 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:21.765865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:21.766085 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:21.766265 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:21.766269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.766867 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.766868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:21.767593 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:21.767594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.767924 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.767925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:21.768060 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:21.768553 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:21.768561 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:21.770126 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:21.771682 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:21.771688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:21.771690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:21.771889 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:21.772088 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:21.772089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:21.772090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:21.772190 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:21.772422 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:21.772428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:21.772551 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 19:00:21.772641 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 19:00:21.772775 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260925 19:00:21.773448 6285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38331
I20260925 19:00:21.773530 6285 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:21.773705 6285 heartbeater.cc:511] Master 127.0.98.62:38331 requested a full tablet report, sending...
I20260925 19:00:21.774082 5960 ts_manager.cc:194] Registered new tserver with Master: ab4aa8afcf4f40ecb5713cf104c0f346 (127.0.98.2:33343)
I20260925 19:00:21.774468 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36299
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:00:21.790812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38331
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:21.866361 6290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:21.866536 6290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:21.866556 6290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:21.867518 6290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:21.867550 6290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:21.867573 6290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:21.869242 6290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:21.869469 6290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:21.869740 6290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:21.874301 6298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:21.874336 6296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:21.874490 6290 server_base.cc:1034] running on GCE node
W20260925 19:00:21.874346 6295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:21.874747 6290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:21.874977 6290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:21.876147 6290 hybrid_clock.cc:648] HybridClock initialized: now 1790362821876107 us; error 37 us; skew 500 ppm
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.878455 6290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:00:21.878841 6290 webserver.cc:460] Webserver started at http://127.0.98.3:35165/ using document root <none> and password file <none>
I20260925 19:00:21.879046 6290 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:21.879106 6290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:21.879241 6290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:21.880236 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "9023e40df3fc494687e90e62130212cb"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "9af12731e2a60208b4cda5e05d113f2e"
server_key_iv: "7bc3c8b54e4089b82507326cad0f6e49"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.880581 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9023e40df3fc494687e90e62130212cb"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "9af12731e2a60208b4cda5e05d113f2e"
server_key_iv: "7bc3c8b54e4089b82507326cad0f6e49"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.881803 6290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:21.882769 6305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.882947 6290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.883042 6290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "9023e40df3fc494687e90e62130212cb"
format_stamp: "Formatted at 2026-09-25 19:00:21 on dist-test-slave-49bz"
server_key: "9af12731e2a60208b4cda5e05d113f2e"
server_key_iv: "7bc3c8b54e4089b82507326cad0f6e49"
server_key_version: "encryptionkey@0"
I20260925 19:00:21.883122 6290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:21.906908 6290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:21.908522 6290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:21.908674 6290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:21.908890 6290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:21.909171 6290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:21.909220 6290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.909253 6290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:21.909279 6290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:21.915764 6290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44383
I20260925 19:00:21.915845 6418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44383 every 8 connection(s)
I20260925 19:00:21.916168 6290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:21.924865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 6290
I20260925 19:00:21.924942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:00:21.925055 392 external_mini_cluster.cc:1428] Setting key b0db0d1bc88c28229ee78fca773b1504
I20260925 19:00:21.927217 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:21.917195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40451 (local address 127.0.98.62:38331)
0925 19:00:21.917263 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:21.917265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:21.917448 (+ 183us) server_negotiation.cc:407] Connection header received
0925 19:00:21.918703 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:21.918705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:21.918717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:21.918775 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:21.919631 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.920457 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.921709 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.922018 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.922091 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:21.924331 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:21.924332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:21.924332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:21.924338 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:21.925837 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.926238 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.926239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.926239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.926317 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.926552 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.926553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.926554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.926659 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:21.926723 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:21.927014 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 19:00:21.927091 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:00:21.927420 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:21.917081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38331 (local address 127.0.98.3:40451)
0925 19:00:21.917306 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:21.917350 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:21.918587 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:21.918840 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:21.918844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:21.919234 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:21.919491 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:21.919498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.920546 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.920547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:21.921614 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:21.921616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.922106 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.922107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:21.922283 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:21.922853 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:21.922863 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:21.924256 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:21.925924 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:21.925929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:21.925930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:21.926152 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:21.926385 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:21.926386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:21.926386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:21.926491 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:21.926724 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:21.926730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:21.926874 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 19:00:21.926994 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 19:00:21.927164 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
I20260925 19:00:21.927805 6419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38331
I20260925 19:00:21.927904 6419 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:21.928068 6419 heartbeater.cc:511] Master 127.0.98.62:38331 requested a full tablet report, sending...
I20260925 19:00:21.928457 5960 ts_manager.cc:194] Registered new tserver with Master: 9023e40df3fc494687e90e62130212cb (127.0.98.3:44383)
I20260925 19:00:21.928787 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40451
I20260925 19:00:21.936858 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/test-admin.keytab.
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.951246 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 19:00:21.952775 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:00:21.952030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.0.98.1:35761)
0925 19:00:21.952182 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:21.952186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:21.952203 (+ 17us) server_negotiation.cc:407] Connection header received
0925 19:00:21.952226 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:21.952227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:21.952240 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:21.952303 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:21.952679 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35262: BlockingRecv error: recv got EOF from 127.0.0.1:35262 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
W20260925 19:00:21.952884 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5cc)
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.953873 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.954825 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 19:00:21.962119 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:21.955308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.0.98.1:35761)
0925 19:00:21.955363 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:21.955365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:21.955413 (+ 48us) server_negotiation.cc:407] Connection header received
0925 19:00:21.955484 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:21.955485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:21.955495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:21.955549 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:21.955940 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.956634 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.957350 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.957661 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.957747 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:21.959650 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:21.959657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:21.959657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:21.959664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:21.961011 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.961345 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.961348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.961349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.961403 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.961609 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.961610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.961610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.961734 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:21.961895 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:21.961946 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 19:00:21.962013 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 19:00:21.963609 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:00:21.963195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.0.98.1:35761)
0925 19:00:21.963222 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:21.963224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:21.963276 (+ 52us) server_negotiation.cc:407] Connection header received
0925 19:00:21.963312 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:21.963313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:21.963320 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:21.963361 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:21.963562 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35284: BlockingRecv error: recv got EOF from 127.0.0.1:35284 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260925 19:00:21.963814 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5cc)
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.964562 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.965294 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 19:00:21.971469 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:21.965723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.0.98.1:35761)
0925 19:00:21.965764 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:21.965766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:21.965820 (+ 54us) server_negotiation.cc:407] Connection header received
0925 19:00:21.965882 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:21.965883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:21.965891 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:21.965953 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:21.966324 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.966728 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.967361 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.967545 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.967615 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:21.969389 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:21.969390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:21.969390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:21.969395 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:21.970730 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.971001 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.971002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.971002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.971043 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.971177 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.971177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.971177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.971239 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:21.971297 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:21.971376 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 19:00:21.971424 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 19:00:21.972822 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:00:21.972400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.0.98.1:35761)
0925 19:00:21.972438 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:21.972440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:21.972478 (+ 38us) server_negotiation.cc:407] Connection header received
0925 19:00:21.972529 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:21.972529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:21.972535 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:21.972566 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:21.972759 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35310: BlockingRecv error: recv got EOF from 127.0.0.1:35310 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 19:00:21.972898 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790362794046639-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.973675 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:21.974462 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 19:00:21 dist-test-slave-49bz krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 19:00:21.980225 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:21.974855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.0.98.1:35761)
0925 19:00:21.974901 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:21.974902 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:21.974949 (+ 47us) server_negotiation.cc:407] Connection header received
0925 19:00:21.974991 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:21.974992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:21.974998 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:21.975036 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:21.975356 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.975732 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.976412 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:21.976564 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:21.976611 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:21.978357 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:21.978358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:21.978358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:21.978363 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:21.979356 (+ 993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.979647 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.979648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.979648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.979692 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:21.979849 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:21.979849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:21.979850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:21.979926 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:21.979988 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:21.980070 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 19:00:21.980116 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 19:00:21.980825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 6022
I20260925 19:00:21.985193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 6156
I20260925 19:00:21.989001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 6290
I20260925 19:00:21.992856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 5928
2026-09-25T19:00:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1769 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:22 dist-test-slave-49bz krb5kdc[6439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:22 dist-test-slave-49bz krb5kdc[6439](info): set up 2 sockets
Sep 25 19:00:22 dist-test-slave-49bz krb5kdc[6439](info): commencing operation
krb5kdc: starting...
W20260925 19:00:22.978202 392 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 19:00:23.084971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41805
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:23.159786 6452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:23.160007 6452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:23.160041 6452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:23.161152 6452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:23.161209 6452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:23.161231 6452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:23.161250 6452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:23.161278 6452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:23.162637 6452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:23.162894 6452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:23.163095 6452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:23.167404 6457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:23.167498 6452 server_base.cc:1034] running on GCE node
W20260925 19:00:23.167598 6458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:23.167420 6460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:23.167876 6452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:23.168134 6452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:23.169301 6452 hybrid_clock.cc:648] HybridClock initialized: now 1790362823169271 us; error 47 us; skew 500 ppm
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:23.171698 6452 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 19:00:23.172142 6452 webserver.cc:460] Webserver started at http://127.0.98.62:42005/ using document root <none> and password file <none>
I20260925 19:00:23.172349 6452 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:23.172411 6452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:23.172561 6452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:23.173565 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "b5986923f01141dfa7d222038a24e3c1"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "1c9174876ca352271065beaaef6b33d2"
server_key_iv: "10295a84086c7bb0c3c2a494cb3695f9"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.173920 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5986923f01141dfa7d222038a24e3c1"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "1c9174876ca352271065beaaef6b33d2"
server_key_iv: "10295a84086c7bb0c3c2a494cb3695f9"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.175238 6452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:23.175887 6467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.176043 6452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.176131 6452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "b5986923f01141dfa7d222038a24e3c1"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "1c9174876ca352271065beaaef6b33d2"
server_key_iv: "10295a84086c7bb0c3c2a494cb3695f9"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.176222 6452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:23.191538 6452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:23.193671 6452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:23.193846 6452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:23.197625 6452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41805
I20260925 19:00:23.197671 6519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41805 every 8 connection(s)
I20260925 19:00:23.198040 6452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:23.198645 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 6452
I20260925 19:00:23.198760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:23.198936 392 external_mini_cluster.cc:1428] Setting key 36bb5ead4689780d3a4f9480c54119f8
I20260925 19:00:23.199072 6520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:23.202488 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1: Bootstrap starting.
I20260925 19:00:23.203107 6520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:23.203400 6520 log.cc:824] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:23.204111 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1: No bootstrap required, opened a new log
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:23.205915 6520 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } }
I20260925 19:00:23.206074 6520 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:23.206118 6520 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5986923f01141dfa7d222038a24e3c1, State: Initialized, Role: FOLLOWER
I20260925 19:00:23.206284 6520 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } }
I20260925 19:00:23.206369 6520 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:23.206403 6520 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:23.206442 6520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:23.207047 6520 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } }
I20260925 19:00:23.207158 6520 leader_election.cc:302] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5986923f01141dfa7d222038a24e3c1; no voters:
I20260925 19:00:23.207350 6520 leader_election.cc:288] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:23.207376 6525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:23.207486 6525 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 1 LEADER]: Becoming Leader. State: Replica: b5986923f01141dfa7d222038a24e3c1, State: Running, Role: LEADER
I20260925 19:00:23.207564 6525 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } }
I20260925 19:00:23.207607 6520 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:23.207893 6526 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5986923f01141dfa7d222038a24e3c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } }
I20260925 19:00:23.208125 6527 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5986923f01141dfa7d222038a24e3c1. Latest consensus state: current_term: 1 leader_uuid: "b5986923f01141dfa7d222038a24e3c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } }
I20260925 19:00:23.208211 6526 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:23.208225 6527 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:23.208536 6534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:23.208976 6534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:23.209861 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.199455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.0.98.62:41805)
0925 19:00:23.199605 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.199614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.199644 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:00:23.200148 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.200157 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.200224 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.200373 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.200928 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.202192 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.203003 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.203602 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.203726 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.205916 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.205921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.205921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.205933 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.207971 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.208395 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.208397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.208398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.208518 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.208750 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.208751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.208751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.209096 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.209252 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.209489 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.209620 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260925 19:00:23.210589 6534 catalog_manager.cc:1347] Generated new cluster ID: 80fea2c92a1a4e88b2ef5600d515a0be
I20260925 19:00:23.210644 6534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:00:23.219117 6534 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:23.219640 6534 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:23.228710 6534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1: Generated new TSK 0
I20260925 19:00:23.228866 6534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 19:00:23.259377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:23.364683 6546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:23.364895 6546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:23.364933 6546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:23.366061 6546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:23.366127 6546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:23.366183 6546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:23.367666 6546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:23.367914 6546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:23.368151 6546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:23.373629 6552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:23.373636 6554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:23.373976 6546 server_base.cc:1034] running on GCE node
W20260925 19:00:23.373636 6551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:23.374248 6546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:23.374521 6546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:23.375677 6546 hybrid_clock.cc:648] HybridClock initialized: now 1790362823375671 us; error 38 us; skew 500 ppm
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:23.377952 6546 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 19:00:23.378427 6546 webserver.cc:460] Webserver started at http://127.0.98.1:38317/ using document root <none> and password file <none>
I20260925 19:00:23.378648 6546 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:23.378710 6546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:23.378846 6546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:23.379889 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "08b3a7016f664931adc2f06fd7aa1b28"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "5b97f5fbdc2bbb4191866dbb61d87679"
server_key_iv: "aee3145c9d7504891a962fce776969d5"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.380241 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08b3a7016f664931adc2f06fd7aa1b28"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "5b97f5fbdc2bbb4191866dbb61d87679"
server_key_iv: "aee3145c9d7504891a962fce776969d5"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.381536 6546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:00:23.382339 6561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.382522 6546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:23.382616 6546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "08b3a7016f664931adc2f06fd7aa1b28"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "5b97f5fbdc2bbb4191866dbb61d87679"
server_key_iv: "aee3145c9d7504891a962fce776969d5"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.382697 6546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:23.419269 6546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:23.421108 6546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:23.421260 6546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:23.421510 6546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:23.421810 6546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:23.421864 6546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.421898 6546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:23.421943 6546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.427372 6546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39433
I20260925 19:00:23.427419 6674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39433 every 8 connection(s)
I20260925 19:00:23.427700 6546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:23.434659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 6546
I20260925 19:00:23.434762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:23.434970 392 external_mini_cluster.cc:1428] Setting key 71bddfd1f601916bbbac47914bf25c53
I20260925 19:00:23.437031 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.428468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46389 (local address 127.0.98.62:41805)
0925 19:00:23.428571 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.428573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.428736 (+ 163us) server_negotiation.cc:407] Connection header received
0925 19:00:23.429248 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.429250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.429262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.429310 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.430240 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.430780 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.431772 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.431983 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.432081 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.434089 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.434090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.434091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.434097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.435694 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.436079 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.436080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.436080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.436141 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.436377 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.436378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.436378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.436493 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.436575 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.436856 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.436936 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 19:00:23.437234 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.428354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:46389)
0925 19:00:23.428599 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:23.428648 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:23.429141 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:23.429383 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:23.429386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:23.429940 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:23.430119 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.430124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.430899 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.430900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.431671 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.431673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.432093 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.432094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.432208 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.432631 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:23.432639 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:23.433916 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:23.435779 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.435791 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.435793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.435992 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.436204 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.436205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.436206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.436304 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.436570 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:23.436576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:23.436720 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 19:00:23.436836 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 19:00:23.436985 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":96,"threads_started":1}
I20260925 19:00:23.437667 6675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41805
I20260925 19:00:23.437752 6675 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:23.437932 6675 heartbeater.cc:511] Master 127.0.98.62:41805 requested a full tablet report, sending...
I20260925 19:00:23.438442 6484 ts_manager.cc:194] Registered new tserver with Master: 08b3a7016f664931adc2f06fd7aa1b28 (127.0.98.1:39433)
I20260925 19:00:23.439010 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46389
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 19:00:23.457273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:23.534058 6680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:23.534235 6680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:23.534272 6680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:23.535272 6680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:23.535302 6680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:23.535327 6680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:23.536597 6680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:23.536765 6680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:23.536926 6680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:23.541524 6686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:23.541551 6688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:23.541509 6685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:23.541762 6680 server_base.cc:1034] running on GCE node
I20260925 19:00:23.541929 6680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:23.542148 6680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:23.543303 6680 hybrid_clock.cc:648] HybridClock initialized: now 1790362823543280 us; error 38 us; skew 500 ppm
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:23.545576 6680 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 19:00:23.545964 6680 webserver.cc:460] Webserver started at http://127.0.98.2:41995/ using document root <none> and password file <none>
I20260925 19:00:23.546183 6680 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:23.546223 6680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:23.546383 6680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:23.547320 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "3b3357daaf754dee986c0af6cbdc6f1f"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "09d63029d7c6dc3a31816330f2eee84d"
server_key_iv: "9022b16ef1d43eced045536dd8058a77"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.547654 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3b3357daaf754dee986c0af6cbdc6f1f"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "09d63029d7c6dc3a31816330f2eee84d"
server_key_iv: "9022b16ef1d43eced045536dd8058a77"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.548883 6680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:00:23.549577 6695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.549764 6680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:23.549860 6680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "3b3357daaf754dee986c0af6cbdc6f1f"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "09d63029d7c6dc3a31816330f2eee84d"
server_key_iv: "9022b16ef1d43eced045536dd8058a77"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.549943 6680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:23.564591 6680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:23.566375 6680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:23.566533 6680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:23.566741 6680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:23.567027 6680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:23.567078 6680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.567117 6680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:23.567145 6680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.572546 6680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45917
I20260925 19:00:23.572609 6808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45917 every 8 connection(s)
I20260925 19:00:23.572876 6680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:23.580865 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.573535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45473 (local address 127.0.98.62:41805)
0925 19:00:23.573612 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.573615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.573764 (+ 149us) server_negotiation.cc:407] Connection header received
0925 19:00:23.574166 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.574166 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.574175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.574224 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.575089 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.575565 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.576444 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.576611 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.576657 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.578573 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.578574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.578574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.578580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.579847 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.580182 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.580183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.580183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.580231 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.580423 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.580423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.580424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.580516 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.580564 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.580747 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.580807 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:00:23.581012 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.573502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:45473)
0925 19:00:23.573656 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:23.573687 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:23.574094 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:23.574329 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:23.574333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:23.574861 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:23.575019 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.575023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.575637 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.575637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.576378 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.576379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.576674 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.576675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.576761 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.577133 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:23.577141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:23.578481 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:23.579913 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.579918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.579920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.580115 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.580285 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.580286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.580286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.580371 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.580554 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:23.580557 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:23.580649 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 19:00:23.580718 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 19:00:23.580821 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260925 19:00:23.581261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 6680
I20260925 19:00:23.581385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:00:23.581492 6809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41805
I20260925 19:00:23.581597 6809 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:23.581604 392 external_mini_cluster.cc:1428] Setting key 23fc1a03fdecf6101bab491ad8c4c267
I20260925 19:00:23.581753 6809 heartbeater.cc:511] Master 127.0.98.62:41805 requested a full tablet report, sending...
I20260925 19:00:23.582125 6483 ts_manager.cc:194] Registered new tserver with Master: 3b3357daaf754dee986c0af6cbdc6f1f (127.0.98.2:45917)
I20260925 19:00:23.582620 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45473
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 19:00:23.602923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:23.680514 6814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:23.680696 6814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:23.680718 6814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:23.681880 6814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:23.681926 6814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:23.681979 6814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:23.683498 6814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:23.683718 6814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:23.683930 6814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:00:23.688738 6814 server_base.cc:1034] running on GCE node
W20260925 19:00:23.688691 6820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:23.688680 6822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:23.688704 6819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:23.689033 6814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:23.689253 6814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:23.690404 6814 hybrid_clock.cc:648] HybridClock initialized: now 1790362823690398 us; error 40 us; skew 500 ppm
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:23.692731 6814 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 19:00:23.693142 6814 webserver.cc:460] Webserver started at http://127.0.98.3:45075/ using document root <none> and password file <none>
I20260925 19:00:23.693370 6814 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:23.693437 6814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:23.693580 6814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:23.694743 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "3a933feeabe44a0db2f6dda09fb121b9"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "9d7eab26aa76f386d50e25c5fcaae4e1"
server_key_iv: "d9950716b54e2017db66bcf0507cdaa0"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.695127 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3a933feeabe44a0db2f6dda09fb121b9"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "9d7eab26aa76f386d50e25c5fcaae4e1"
server_key_iv: "d9950716b54e2017db66bcf0507cdaa0"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.696460 6814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:00:23.697151 6829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.697332 6814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:23.697417 6814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "3a933feeabe44a0db2f6dda09fb121b9"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "9d7eab26aa76f386d50e25c5fcaae4e1"
server_key_iv: "d9950716b54e2017db66bcf0507cdaa0"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.697499 6814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:23.719611 6814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:23.721921 6814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:23.722074 6814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:23.722342 6814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:23.722648 6814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:23.722700 6814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.722733 6814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:23.722760 6814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.728866 6814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37301
I20260925 19:00:23.728896 6942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37301 every 8 connection(s)
I20260925 19:00:23.729194 6814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:23.737002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 6814
I20260925 19:00:23.737090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:00:23.737277 392 external_mini_cluster.cc:1428] Setting key b754810c805cd9acff240fefd680cecb
I20260925 19:00:23.738160 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.729841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52665 (local address 127.0.98.62:41805)
0925 19:00:23.729887 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.729888 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.730136 (+ 248us) server_negotiation.cc:407] Connection header received
0925 19:00:23.730698 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.730699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.730712 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.730775 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.731679 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.732167 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.733126 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.733313 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.733397 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.735320 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.735322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.735322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.735328 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.736673 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.737133 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.737134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.737134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.737196 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.737474 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.737475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.737476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.737593 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.737644 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.737990 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.738065 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:00:23.738394 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.729839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:52665)
0925 19:00:23.730007 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:23.730062 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:23.730595 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:23.730844 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:23.730848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:23.731414 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:23.731584 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.731588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.732261 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.732262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.733023 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.733025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.733405 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.733406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.733528 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.733926 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:23.733934 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:23.735227 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:23.736753 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.736757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.736759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.737051 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.737270 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.737271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.737271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.737399 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.737700 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:23.737705 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:23.737850 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 19:00:23.737959 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 19:00:23.738128 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260925 19:00:23.738749 6943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41805
I20260925 19:00:23.738849 6943 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:23.739027 6943 heartbeater.cc:511] Master 127.0.98.62:41805 requested a full tablet report, sending...
I20260925 19:00:23.739351 6483 ts_manager.cc:194] Registered new tserver with Master: 3a933feeabe44a0db2f6dda09fb121b9 (127.0.98.3:37301)
I20260925 19:00:23.739725 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52665
I20260925 19:00:23.749177 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260925 19:00:23.771122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39997
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41805,127.0.98.61:39997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:23.847293 6948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:23.847474 6948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:23.847494 6948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:23.848613 6948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:23.848652 6948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:23.848666 6948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:23.848678 6948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:23.848718 6948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:23.850201 6948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41805,127.0.98.61:39997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
W20260925 19:00:23.850347 6948 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:41805,127.0.98.61:39997'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 19:00:23.850487 6948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:23.850694 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:00:23.855156 6948 server_base.cc:1034] running on GCE node
W20260925 19:00:23.855141 6954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:23.855134 6956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:23.855158 6953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:23.855515 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:23.855784 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:23.856946 6948 hybrid_clock.cc:648] HybridClock initialized: now 1790362823856927 us; error 32 us; skew 500 ppm
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:23.859361 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 19:00:23.859802 6948 webserver.cc:460] Webserver started at http://127.0.98.61:39193/ using document root <none> and password file <none>
I20260925 19:00:23.860028 6948 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:23.860092 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:23.860237 6948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:23.861227 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data/instance:
uuid: "3580c5192ad74c98bbeb4254f62d1fac"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "2589af800994e673b981743750f6e583"
server_key_iv: "4687ed53c35713715d8f7bc30ca35c26"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.861577 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal/instance:
uuid: "3580c5192ad74c98bbeb4254f62d1fac"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "2589af800994e673b981743750f6e583"
server_key_iv: "4687ed53c35713715d8f7bc30ca35c26"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.862918 6948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:00:23.863669 6963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.863855 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:23.863951 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal
uuid: "3580c5192ad74c98bbeb4254f62d1fac"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "2589af800994e673b981743750f6e583"
server_key_iv: "4687ed53c35713715d8f7bc30ca35c26"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.864024 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:23.877121 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:23.878852 6948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:23.878996 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:23.882807 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39997
I20260925 19:00:23.882851 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39997 every 8 connection(s)
I20260925 19:00:23.883172 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
I20260925 19:00:23.884011 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:23.885017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 6948
I20260925 19:00:23.885619 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 19:00:23.894778 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.886318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.0.98.62:41805)
0925 19:00:23.886371 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.886373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.886570 (+ 197us) server_negotiation.cc:407] Connection header received
0925 19:00:23.886636 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.886637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.886647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.886699 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.887363 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.888109 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.889278 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.889611 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.889680 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.891836 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.891837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.891838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.891844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.893404 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.893780 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.893780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.893781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.893824 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.894069 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.894070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.894070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.894279 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.894338 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.894627 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.894682 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:00:23.894964 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.886214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:45186)
0925 19:00:23.886454 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:23.886487 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:23.886567 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:23.886718 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:23.886720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:23.887063 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:23.887275 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.887281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.888204 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.888205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.889197 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.889199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.889691 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.889692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.889796 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.890227 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:23.890234 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:23.891732 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:23.893455 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.893460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.893462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.893711 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.893884 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.893885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.893885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.894005 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.894328 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:23.894331 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:23.894475 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 19:00:23.894595 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 19:00:23.894740 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":181,"threads_started":1}
I20260925 19:00:23.895287 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.885639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.0.98.61:39997)
0925 19:00:23.885810 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.885819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.885851 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:00:23.886444 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.886450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.886493 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.886622 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.887336 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.888388 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.889343 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.889823 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.889878 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.892274 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.892282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.892282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.892289 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.893904 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.894313 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.894316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.894317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.894371 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.894546 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.894547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.894547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.894681 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.894806 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.895102 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.895179 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260925 19:00:23.895758 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 19:00:23.901667 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.895972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39997 (local address 127.0.0.1:56804)
0925 19:00:23.896019 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:23.896028 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:23.896067 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:23.896180 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:23.896181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:23.896307 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:23.896336 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.896337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.896911 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.896911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.897350 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.897352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.897703 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.897703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.897757 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.898048 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:23.898051 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:23.899302 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:23.900839 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.900840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.900840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.901008 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.901215 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.901215 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.901216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.901271 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.901445 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:23.901446 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:23.901477 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 19:00:23.901526 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:00:23.901596 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260925 19:00:23.901713 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.896022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.0.98.61:39997)
0925 19:00:23.896051 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.896053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.896060 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:00:23.896119 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.896120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.896128 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.896177 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.896413 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.896832 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.897435 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.897626 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.897702 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.899412 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.899413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.899414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.899420 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.900772 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.901119 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.901120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.901120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.901162 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.901320 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.901320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.901320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.901397 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.901439 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.901538 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.901634 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":15}
I20260925 19:00:23.902906 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: Bootstrap starting.
I20260925 19:00:23.903662 7016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:23.903944 7016 log.cc:824] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:23.904627 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: No bootstrap required, opened a new log
I20260925 19:00:23.905917 7016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } }
I20260925 19:00:23.906060 7016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:23.906106 7016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3580c5192ad74c98bbeb4254f62d1fac, State: Initialized, Role: FOLLOWER
I20260925 19:00:23.906229 7016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } }
I20260925 19:00:23.906590 7016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:23.906574 7022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } } }
I20260925 19:00:23.906673 7022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:00:23.908170 7033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:00:23.908257 7033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:00:23.909584 6948 master_runner.cc:428] Detected that this master 3580c5192ad74c98bbeb4254f62d1fac is joining an existing cluster
I20260925 19:00:23.909652 6948 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:39997
I20260925 19:00:23.917899 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.910651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.0.98.62:41805)
0925 19:00:23.910696 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.910698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.911047 (+ 349us) server_negotiation.cc:407] Connection header received
0925 19:00:23.911114 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.911115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.911125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.911172 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.911606 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.912310 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.913408 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.913728 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.913781 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.914816 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.914817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.914818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.914825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.916620 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.917144 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.917146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.917146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.917202 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.917399 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.917399 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.917400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.917506 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.917546 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.917741 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.917824 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:00:23.918300 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.910621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:45188)
0925 19:00:23.911002 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:23.911010 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:23.911066 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:23.911184 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:23.911185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:23.911439 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:23.911527 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.911528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.912382 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.912383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.913300 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.913303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.913855 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.913856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.913940 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.914280 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:23.914284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:23.914706 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:23.916719 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.916720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.916720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.917058 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.917273 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.917273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.917274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.917334 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.917591 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:23.917592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:23.917643 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 19:00:23.918071 (+ 428us) client_negotiation.cc:240] Negotiation successful
0925 19:00:23.918171 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":116,"threads_started":1}
I20260925 19:00:23.918982 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.910717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.0.98.61:39997)
0925 19:00:23.910750 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.910751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.910887 (+ 136us) server_negotiation.cc:407] Connection header received
0925 19:00:23.910979 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.910980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.910994 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.911060 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.911457 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.912153 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.913205 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.913455 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.913494 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.914799 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.914800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.914800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.914806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.917142 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.917568 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.917569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.917569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.917637 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.917880 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.917881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.917881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.918171 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.918242 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.918788 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.918880 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:00:23.919471 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.910577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39997 (local address 127.0.0.1:56820)
0925 19:00:23.910816 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:23.910826 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:23.910884 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:23.911071 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:23.911072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:23.911325 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:23.911385 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.911386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.912227 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.912228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.913114 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.913117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.913569 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.913569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.913634 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.914085 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:23.914089 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:23.914694 (+ 605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:23.917216 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.917217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.917217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.917476 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.917719 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.917720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.917720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.917800 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.918336 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:23.918336 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:23.918632 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 19:00:23.919194 (+ 562us) client_negotiation.cc:240] Negotiation successful
0925 19:00:23.919334 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 19:00:23.927981 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.920957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39997 (local address 127.0.0.1:56832)
0925 19:00:23.921103 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:23.921141 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:23.921227 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:23.921377 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:23.921380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:23.921735 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:23.921782 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.921783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.922646 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.922647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.923212 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.923213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.923657 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.923658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.923707 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.924123 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:23.924127 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:23.925461 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:23.927042 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.927048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.927049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.927230 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.927440 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.927441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.927441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.927492 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.927721 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:23.927726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:23.927766 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 19:00:23.927830 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:00:23.927911 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260925 19:00:23.928004 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.921006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.0.98.61:39997)
0925 19:00:23.921080 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.921081 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.921217 (+ 136us) server_negotiation.cc:407] Connection header received
0925 19:00:23.921292 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.921292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.921302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.921372 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.921856 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.922571 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.923290 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.923585 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.923653 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.925536 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.925537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.925537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.925543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.926945 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.927317 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.927319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.927319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.927371 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.927555 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.927556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.927556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.927664 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.927725 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.927863 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.927926 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:23.928474 6483 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:39997: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 39997
}
attrs {
promote: true
}
}
dest_uuid: "b5986923f01141dfa7d222038a24e3c1"
cas_config_opid_index: -1
I20260925 19:00:23.928769 6483 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } }
I20260925 19:00:23.929183 7043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3580c5192ad74c98bbeb4254f62d1fac (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } } }
I20260925 19:00:23.929509 6990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:23.930233 7043 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:39997
I20260925 19:00:23.930455 6948 master.cc:537] Master@127.0.98.61:39997 shutting down...
I20260925 19:00:23.930477 6990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Refusing update from remote peer b5986923f01141dfa7d222038a24e3c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 19:00:23.930500 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5986923f01141dfa7d222038a24e3c1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } } }
I20260925 19:00:23.930562 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:23.930883 7042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:00:23.931599 6469 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 19:00:23.932685 6948 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:23.932772 6948 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:23.932817 6948 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: stopping tablet replica
I20260925 19:00:23.934549 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.934430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39997 (local address 127.0.0.1:56848)
0925 19:00:23.934491 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:23.934523 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 19:00:23.934636 6469 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 -> Peer 3580c5192ad74c98bbeb4254f62d1fac (127.0.98.61:39997): Couldn't send request to peer 3580c5192ad74c98bbeb4254f62d1fac. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39997: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 19:00:23.945016 6948 master.cc:552] Master@127.0.98.61:39997 shutdown complete.
I20260925 19:00:23.945061 6948 master_runner.cc:305] Clearing existing system tablet
I20260925 19:00:23.945181 6948 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:23.945225 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:23.945878 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:23.946426 7047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.946570 6948 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:23.946645 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:23.946677 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal
uuid: "3580c5192ad74c98bbeb4254f62d1fac"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "2589af800994e673b981743750f6e583"
server_key_iv: "4687ed53c35713715d8f7bc30ca35c26"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.947216 6948 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:23.948412 6948 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 19:00:23.948484 6948 log.cc:1196] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 19:00:23.948663 6948 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: Deleting consensus metadata
I20260925 19:00:23.948781 6948 master_runner.cc:315] Copying system tablet from 127.0.98.62:41805
I20260925 19:00:23.949348 6948 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41805
I20260925 19:00:23.955636 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.949586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:45202)
0925 19:00:23.949691 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:23.949700 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:23.949763 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:23.949927 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:23.949928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:23.950072 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:23.950435 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.950437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.951376 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.951377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.952095 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:23.952097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.952525 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.952525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:23.952567 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.953016 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:23.953021 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:23.953302 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:23.954828 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.954828 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.954829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.954969 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.955164 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:23.955165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:23.955165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:23.955210 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:23.955435 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:23.955436 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:23.955466 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 19:00:23.955510 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 19:00:23.955573 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":59,"threads_started":1}
I20260925 19:00:23.955669 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.949639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.0.98.62:41805)
0925 19:00:23.949682 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.949684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.949754 (+ 70us) server_negotiation.cc:407] Connection header received
0925 19:00:23.949820 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.949820 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.949830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.949918 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.950507 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.951284 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.952165 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.952454 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.952523 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:23.953371 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:23.953372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:23.953372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:23.953378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:23.954754 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.955036 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.955037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.955037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.955106 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:23.955270 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:23.955271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:23.955271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:23.955380 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:23.955439 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:23.955531 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 19:00:23.955593 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:00:23.956069 6504 tablet_copy_service.cc:140] P b5986923f01141dfa7d222038a24e3c1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3580c5192ad74c98bbeb4254f62d1fac ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45202)
I20260925 19:00:23.956164 6504 tablet_copy_service.cc:161] P b5986923f01141dfa7d222038a24e3c1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3580c5192ad74c98bbeb4254f62d1fac at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45202: session id = 3580c5192ad74c98bbeb4254f62d1fac-00000000000000000000000000000000
I20260925 19:00:23.956914 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 19:00:23.957414 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 19:00:23.958137 6948 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: tablet copy: Starting download of 0 data blocks...
I20260925 19:00:23.958328 6948 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: tablet copy: Starting download of 1 WAL segments...
I20260925 19:00:23.959203 6948 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 19:00:23.959931 6504 tablet_copy_service.cc:342] P b5986923f01141dfa7d222038a24e3c1: Request end of tablet copy session 3580c5192ad74c98bbeb4254f62d1fac-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45202
I20260925 19:00:23.959987 6504 tablet_copy_service.cc:433] P b5986923f01141dfa7d222038a24e3c1: ending tablet copy session 3580c5192ad74c98bbeb4254f62d1fac-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3580c5192ad74c98bbeb4254f62d1fac
I20260925 19:00:23.962124 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:23.963212 7055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:23.963238 7057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:23.963380 7054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:23.966416 6948 server_base.cc:1034] running on GCE node
I20260925 19:00:23.966526 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:23.966643 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:23.967773 6948 hybrid_clock.cc:648] HybridClock initialized: now 1790362823967759 us; error 28 us; skew 500 ppm
Sep 25 19:00:23 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:23.969007 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 19:00:23.969220 6948 webserver.cc:460] Webserver started at http://127.0.98.61:33021/ using document root <none> and password file <none>
I20260925 19:00:23.969323 6948 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:23.969355 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:23.970021 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.970556 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.970690 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:23.970742 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal
uuid: "3580c5192ad74c98bbeb4254f62d1fac"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "2589af800994e673b981743750f6e583"
server_key_iv: "4687ed53c35713715d8f7bc30ca35c26"
server_key_version: "encryptionkey@0"
I20260925 19:00:23.970798 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:23.982239 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:23.983731 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:23.986802 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39997
I20260925 19:00:23.986992 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39997 every 8 connection(s)
I20260925 19:00:23.987149 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
I20260925 19:00:23.987862 7117 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:00:23.988196 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: Bootstrap starting.
I20260925 19:00:23.990240 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:00:23.990520 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: Bootstrap complete.
I20260925 19:00:23.990720 7117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } }
I20260925 19:00:23.990801 7117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3580c5192ad74c98bbeb4254f62d1fac, State: Initialized, Role: LEARNER
I20260925 19:00:23.990844 7117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } }
I20260925 19:00:23.991009 7117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:23.991006 7120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } } }
I20260925 19:00:23.991112 7120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: This master's current role is: LEARNER
I20260925 19:00:23.992383 7131 catalog_manager.cc:1259] Loaded cluster ID: 80fea2c92a1a4e88b2ef5600d515a0be
I20260925 19:00:23.992425 7131 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: loading cluster ID for follower catalog manager: success
I20260925 19:00:23.993315 7131 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: acquiring CA information for follower catalog manager: success
I20260925 19:00:23.993571 7131 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:00:24.003100 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:23.996697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.0.98.61:39997)
0925 19:00:23.996788 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:23.996790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:23.996803 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:00:23.996827 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:23.996827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:23.996835 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:23.996892 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:23.997485 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.998032 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.999001 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:23.999174 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:23.999224 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.000637 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.000638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.000638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.000644 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.002036 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.002439 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.002440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.002440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.002488 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.002691 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.002691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.002691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.002770 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.002843 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.002982 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.003033 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":58,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260925 19:00:24.027441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40113
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41805,127.0.98.61:39997,127.0.98.60:40113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:24.110569 7135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:24.110754 7135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:24.110773 7135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:24.111925 7135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:24.111965 7135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:24.111979 7135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:24.111991 7135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:24.112002 7135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:24.113423 7135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41805,127.0.98.61:39997,127.0.98.60:40113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:24.113651 7135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:24.113876 7135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:24.118291 7143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:24.118304 7141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:24.118299 7140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:24.119043 7135 server_base.cc:1034] running on GCE node
I20260925 19:00:24.119247 7135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:24.119537 7135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:24.120682 7135 hybrid_clock.cc:648] HybridClock initialized: now 1790362824120668 us; error 28 us; skew 500 ppm
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:24.123039 7135 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 19:00:24.123500 7135 webserver.cc:460] Webserver started at http://127.0.98.60:44811/ using document root <none> and password file <none>
I20260925 19:00:24.123715 7135 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:24.123786 7135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:24.123939 7135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:24.124926 7135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data/instance:
uuid: "fc5658ea063c468eb33f350dfa9ded66"
format_stamp: "Formatted at 2026-09-25 19:00:24 on dist-test-slave-49bz"
server_key: "9fa3297e612aca3013ebb6461989bb7b"
server_key_iv: "a5109f41be599085eb3284b778aaf0a0"
server_key_version: "encryptionkey@0"
I20260925 19:00:24.125494 7135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal/instance:
uuid: "fc5658ea063c468eb33f350dfa9ded66"
format_stamp: "Formatted at 2026-09-25 19:00:24 on dist-test-slave-49bz"
server_key: "9fa3297e612aca3013ebb6461989bb7b"
server_key_iv: "a5109f41be599085eb3284b778aaf0a0"
server_key_version: "encryptionkey@0"
I20260925 19:00:24.126947 7135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 19:00:24.127627 7150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.127796 7135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:24.127913 7135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal
uuid: "fc5658ea063c468eb33f350dfa9ded66"
format_stamp: "Formatted at 2026-09-25 19:00:24 on dist-test-slave-49bz"
server_key: "9fa3297e612aca3013ebb6461989bb7b"
server_key_iv: "a5109f41be599085eb3284b778aaf0a0"
server_key_version: "encryptionkey@0"
I20260925 19:00:24.127991 7135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:24.145494 7135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:24.147605 7135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:24.147754 7135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:24.151923 7135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40113
I20260925 19:00:24.151947 7202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40113 every 8 connection(s)
I20260925 19:00:24.152298 7135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
I20260925 19:00:24.153213 7203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:24.154685 7203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:24.162246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 7135
I20260925 19:00:24.162374 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.155417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.0.98.62:41805)
0925 19:00:24.155475 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.155477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.155487 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:00:24.155878 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.155879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.155888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.155935 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.156433 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.156902 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.157808 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.157981 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.158026 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.159874 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.159875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.159876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.159882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.161229 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.161549 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.161549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.161550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.161593 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.161770 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.161770 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.161770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.161865 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.161912 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.162185 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.162277 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 19:00:24.162597 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.155231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:45216)
0925 19:00:24.155386 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.155424 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.155804 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.155989 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.155992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.156198 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.156371 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.156374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.156968 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.156969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.157743 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.157744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.158041 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.158041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.158142 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.158529 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.158537 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.159788 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.161295 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.161299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.161300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.161492 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.161641 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.161642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.161642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.161724 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.161939 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.161945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.162051 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.162152 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.162338 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260925 19:00:24.163123 7203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 19:00:24.169776 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.163452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.0.98.61:39997)
0925 19:00:24.163492 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.163494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.163540 (+ 46us) server_negotiation.cc:407] Connection header received
0925 19:00:24.163611 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.163624 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.163637 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.163678 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.163971 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.164469 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.165310 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.165498 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.165547 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.167208 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.167210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.167210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.167236 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.168659 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.169062 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.169063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.169063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.169107 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.169307 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.169308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.169308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.169435 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.169488 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.169651 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.169712 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:00:24.169806 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.163378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39997 (local address 127.0.0.1:56878)
0925 19:00:24.163498 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.163509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.163553 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.163689 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.163689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.163847 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.163890 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.163892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.164553 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.164554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.165192 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.165194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.165558 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.165559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.165619 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.165986 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.165989 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.167134 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.168724 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.168725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.168726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.168994 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.169165 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.169166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.169166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.169236 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.169505 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.169506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.169551 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.169627 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.169712 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 19:00:24.170173 7203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40113 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:00:24.171566 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.162926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.0.98.60:40113)
0925 19:00:24.163046 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.163055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.163069 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:00:24.163086 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.163087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.163097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.163152 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.163830 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.164593 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.165255 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.165575 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.165635 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.168251 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.168257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.168257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.168263 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.170325 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.170772 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.170775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.170775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.170836 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.171036 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.171037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.171038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.171166 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.171324 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.171390 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.171461 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 19:00:24.176509 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.170445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40113 (local address 127.0.0.1:51374)
0925 19:00:24.170488 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.170498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.170531 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.170751 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.170752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.170866 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.170886 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.170887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.171533 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.171534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.172240 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.172242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.172584 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.172585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.172629 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.172945 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.172949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.174019 (+ 1070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.175793 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.175794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.175794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.175947 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.176101 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.176101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.176101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.176146 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.176318 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.176318 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.176348 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.176394 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.176457 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 19:00:24.176668 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.170481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.0.98.60:40113)
0925 19:00:24.170619 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.170620 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.170626 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:00:24.170642 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.170642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.170654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.170721 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.170950 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.171468 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.172310 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.172512 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.172584 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.174089 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.174090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.174090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.174096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.175729 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.176005 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.176005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.176006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.176051 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.176182 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.176182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.176182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.176258 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.176322 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.176408 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.176500 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260925 19:00:24.177740 7203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: Bootstrap starting.
I20260925 19:00:24.178362 7203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:24.178632 7203 log.cc:824] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:24.179333 7203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: No bootstrap required, opened a new log
I20260925 19:00:24.180794 7203 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40113 } }
I20260925 19:00:24.180912 7203 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:24.180951 7203 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5658ea063c468eb33f350dfa9ded66, State: Initialized, Role: FOLLOWER
I20260925 19:00:24.181077 7203 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40113 } }
I20260925 19:00:24.181285 7209 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40113 } } }
I20260925 19:00:24.181371 7203 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:24.181418 7209 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:00:24.183262 7220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:00:24.183320 7220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:00:24.185685 7135 master_runner.cc:428] Detected that this master fc5658ea063c468eb33f350dfa9ded66 is joining an existing cluster
I20260925 19:00:24.185745 7135 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:40113
I20260925 19:00:24.210500 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.187124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.0.98.60:40113)
0925 19:00:24.187160 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.187162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.187277 (+ 115us) server_negotiation.cc:407] Connection header received
0925 19:00:24.187358 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.187359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.187376 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.187403 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.187848 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.188581 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.189626 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.189944 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.189983 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.207046 (+ 17063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.207048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.207048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.207054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.209294 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.209671 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.209673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.209673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.209717 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.209901 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.209901 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.209901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.210031 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.210069 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.210234 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.210406 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:00:24.210933 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.186819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.0.98.62:41805)
0925 19:00:24.186872 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.186873 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.186880 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:00:24.186899 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.186899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.186908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.186962 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.187330 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.188051 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.189148 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.189471 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.189523 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.207592 (+ 18069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.207593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.207593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.207600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.209655 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.210063 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.210064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.210064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.210117 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.210341 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.210342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.210342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.210479 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.210522 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.210747 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.210838 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:00:24.211256 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.187108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40113 (local address 127.0.0.1:51378)
0925 19:00:24.187231 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.187240 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.187296 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.187453 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.187453 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.187694 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.187760 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.187763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.188673 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.188674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.189540 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.189542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.190050 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.190051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.190107 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.190502 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.190505 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.206934 (+ 16429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.209380 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.209381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.209382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.209589 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.209776 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.209776 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.209777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.209840 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.210106 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.210107 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.210141 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.210309 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.210936 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":41,"thread_start_us":74,"threads_started":1}
I20260925 19:00:24.211333 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.186813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.0.98.61:39997)
0925 19:00:24.186853 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.186854 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.187273 (+ 419us) server_negotiation.cc:407] Connection header received
0925 19:00:24.187375 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.187376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.187386 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.187413 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.187848 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.188599 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.189630 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.190296 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.190370 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.207417 (+ 17047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.207418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.207419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.207427 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.210068 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.210522 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.210523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.210524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.210576 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.210774 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.210774 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.210775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.210894 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.210935 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.211145 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.211237 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:00:24.211328 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.186638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:45226)
0925 19:00:24.186764 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.186774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.186823 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.186978 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.186979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.187164 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.187222 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.187224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.188144 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.188145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.189052 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.189055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.189629 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.189630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.189697 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.190152 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.190157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.207497 (+ 17340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.209738 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.209739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.209740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.209974 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.210189 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.210190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.210190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.210240 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.210568 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.210569 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.210627 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.211063 (+ 436us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.211166 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260925 19:00:24.211568 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.186639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39997 (local address 127.0.0.1:56894)
0925 19:00:24.187213 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.187223 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.187296 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.187471 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.187472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.187690 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.187760 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.187763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.188678 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.188679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.189540 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.189542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.190371 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.190372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.190434 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.206844 (+ 16410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.206851 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.207314 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.210148 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.210149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.210150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.210438 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.210647 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.210647 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.210648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.210705 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.210996 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.210996 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.211037 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.211426 (+ 389us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.211511 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":35,"spinlock_wait_cycles":8064,"thread_start_us":98,"threads_started":1}
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 19:00:24.219460 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.213552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.0.98.60:40113)
0925 19:00:24.213589 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.213591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.213598 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:00:24.213629 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.213630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.213637 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.213689 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.213931 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.214432 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.215117 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.215314 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.215398 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.217297 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.217298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.217298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.217304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.218511 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.218849 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.218849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.218850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.218891 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.219057 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.219057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.219057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.219138 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.219183 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.219356 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.219408 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:00:24.219528 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.213490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40113 (local address 127.0.0.1:51390)
0925 19:00:24.213532 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.213542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.213577 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.213692 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.213693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.213832 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.213866 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.213867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.214499 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.214500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.215027 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.215028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.215397 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.215397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.215464 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.215850 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.215854 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.217227 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.218579 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.218580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.218581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.218783 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.218946 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.218947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.218947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.219006 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.219189 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.219190 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.219230 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.219338 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.219434 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 19:00:24.220000 6483 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:40113: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 40113
}
attrs {
promote: true
}
}
dest_uuid: "b5986923f01141dfa7d222038a24e3c1"
cas_config_opid_index: 5
I20260925 19:00:24.220233 6483 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } }
I20260925 19:00:24.220559 7042 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fc5658ea063c468eb33f350dfa9ded66 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
I20260925 19:00:24.221516 7042 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:40113
I20260925 19:00:24.221592 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5986923f01141dfa7d222038a24e3c1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
I20260925 19:00:24.221661 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:24.221848 7177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:24.222512 7135 master.cc:537] Master@127.0.98.60:40113 shutting down...
I20260925 19:00:24.223340 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.220593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39997 (local address 127.0.0.1:56904)
0925 19:00:24.220671 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.220681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.220717 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.220884 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.220885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.220887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:24.220918 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.220919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.221540 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.221542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.222555 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.222558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.223112 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.223113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.223165 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.223183 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.223238 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.223270 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 19:00:24.223332 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.220672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.0.98.61:39997)
0925 19:00:24.220735 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.220737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.220745 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:24.220767 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.220768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.220787 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.220869 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:24.220994 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.221446 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.222638 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.223026 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.223102 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.223173 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.223244 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.223277 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:00:24.223667 7177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [term 1 FOLLOWER]: Refusing update from remote peer b5986923f01141dfa7d222038a24e3c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 19:00:24.223927 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:24.224174 7091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 LEARNER]: Refusing update from remote peer b5986923f01141dfa7d222038a24e3c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 19:00:24.224452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 6546
I20260925 19:00:24.225073 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 19:00:24.225589 7091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fc5658ea063c468eb33f350dfa9ded66 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
W20260925 19:00:24.226477 6471 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 -> Peer fc5658ea063c468eb33f350dfa9ded66 (127.0.98.60:40113): Couldn't send request to peer fc5658ea063c468eb33f350dfa9ded66. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 19:00:24.226892 7120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5986923f01141dfa7d222038a24e3c1. Latest consensus state: current_term: 1 leader_uuid: "b5986923f01141dfa7d222038a24e3c1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
I20260925 19:00:24.226982 7120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: This master's current role is: LEARNER
I20260925 19:00:24.227020 7120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5986923f01141dfa7d222038a24e3c1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: true } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
I20260925 19:00:24.227077 7120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: This master's current role is: LEARNER
I20260925 19:00:24.227680 7043 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1: attempting to promote NON_VOTER 3580c5192ad74c98bbeb4254f62d1fac to VOTER
I20260925 19:00:24.227869 7043 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } }
I20260925 19:00:24.228402 7091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 LEARNER]: Refusing update from remote peer b5986923f01141dfa7d222038a24e3c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 19:00:24.228621 7231 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260925 19:00:24.228696 6471 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 -> Peer fc5658ea063c468eb33f350dfa9ded66 (127.0.98.60:40113): Couldn't send request to peer fc5658ea063c468eb33f350dfa9ded66. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 19:00:24.229193 7042 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3580c5192ad74c98bbeb4254f62d1fac (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
I20260925 19:00:24.229307 7091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3580c5192ad74c98bbeb4254f62d1fac (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
I20260925 19:00:24.230020 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5986923f01141dfa7d222038a24e3c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
I20260925 19:00:24.230087 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:24.230134 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5986923f01141dfa7d222038a24e3c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
I20260925 19:00:24.230172 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:24.230377 7120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5986923f01141dfa7d222038a24e3c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
I20260925 19:00:24.230468 7120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:00:24.231667 7135 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:24.231745 7135 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:24.231807 7135 tablet_replica.cc:330] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: stopping tablet replica
I20260925 19:00:24.232263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39433
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41805
--webserver_port=38317
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41805,127.0.98.61:39997
--tserver_master_addrs=127.0.98.62:41805,127.0.98.61:39997,127.0.98.60:40113 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:00:24.245157 7135 master.cc:552] Master@127.0.98.60:40113 shutdown complete.
I20260925 19:00:24.245208 7135 master_runner.cc:305] Clearing existing system tablet
I20260925 19:00:24.245362 7135 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:24.245409 7135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:24.246119 7135 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:24.246631 7235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.246760 7135 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:24.246805 7135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.246826 7135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal
uuid: "fc5658ea063c468eb33f350dfa9ded66"
format_stamp: "Formatted at 2026-09-25 19:00:24 on dist-test-slave-49bz"
server_key: "9fa3297e612aca3013ebb6461989bb7b"
server_key_iv: "a5109f41be599085eb3284b778aaf0a0"
server_key_version: "encryptionkey@0"
I20260925 19:00:24.247548 7135 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:24.248989 7135 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 19:00:24.249055 7135 log.cc:1196] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 19:00:24.249233 7135 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: Deleting consensus metadata
I20260925 19:00:24.249365 7135 master_runner.cc:315] Copying system tablet from 127.0.98.62:41805
I20260925 19:00:24.249938 7135 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41805
I20260925 19:00:24.256966 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.250292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.0.98.62:41805)
0925 19:00:24.250346 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.250348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.250485 (+ 137us) server_negotiation.cc:407] Connection header received
0925 19:00:24.250533 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.250534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.250545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.250620 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.250963 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.251741 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.252752 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.253055 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.253123 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.254168 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.254170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.254171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.254179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.255828 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.256198 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.256200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.256200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.256258 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.256468 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.256469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.256469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.256588 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.256651 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.256811 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.256882 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:00:24.256970 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.250211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:45234)
0925 19:00:24.250413 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.250423 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.250484 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.250630 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.250630 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.250832 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.250889 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.250891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.251828 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.251830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.252674 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.252676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.253133 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.253135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.253201 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.253698 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.253702 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.254077 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.255908 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.255909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.255909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.256125 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.256329 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.256330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.256330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.256398 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.256657 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.256657 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.256704 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.256780 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.256881 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":150,"threads_started":1}
I20260925 19:00:24.257536 6504 tablet_copy_service.cc:140] P b5986923f01141dfa7d222038a24e3c1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fc5658ea063c468eb33f350dfa9ded66 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45234)
I20260925 19:00:24.257601 6504 tablet_copy_service.cc:161] P b5986923f01141dfa7d222038a24e3c1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fc5658ea063c468eb33f350dfa9ded66 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45234: session id = fc5658ea063c468eb33f350dfa9ded66-00000000000000000000000000000000
I20260925 19:00:24.258152 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 19:00:24.258955 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 19:00:24.260097 7135 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: tablet copy: Starting download of 0 data blocks...
I20260925 19:00:24.260231 7135 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: tablet copy: Starting download of 1 WAL segments...
I20260925 19:00:24.260879 7135 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 19:00:24.261747 6504 tablet_copy_service.cc:342] P b5986923f01141dfa7d222038a24e3c1: Request end of tablet copy session fc5658ea063c468eb33f350dfa9ded66-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45234
I20260925 19:00:24.261798 6504 tablet_copy_service.cc:433] P b5986923f01141dfa7d222038a24e3c1: ending tablet copy session fc5658ea063c468eb33f350dfa9ded66-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fc5658ea063c468eb33f350dfa9ded66
I20260925 19:00:24.265961 7135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:24.267105 7243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:24.267160 7242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:24.271169 7135 server_base.cc:1034] running on GCE node
W20260925 19:00:24.271138 7245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:24.271356 7135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:24.271486 7135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:24.272601 7135 hybrid_clock.cc:648] HybridClock initialized: now 1790362824272609 us; error 43 us; skew 500 ppm
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:24.274058 7135 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 19:00:24.274322 7135 webserver.cc:460] Webserver started at http://127.0.98.60:43977/ using document root <none> and password file <none>
I20260925 19:00:24.274444 7135 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:24.274474 7135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:24.275161 7135 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:24.275730 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.275894 7135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:24.275949 7135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal
uuid: "fc5658ea063c468eb33f350dfa9ded66"
format_stamp: "Formatted at 2026-09-25 19:00:24 on dist-test-slave-49bz"
server_key: "9fa3297e612aca3013ebb6461989bb7b"
server_key_iv: "a5109f41be599085eb3284b778aaf0a0"
server_key_version: "encryptionkey@0"
I20260925 19:00:24.276000 7135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:24.291306 7135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:24.293107 7135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:24.296078 7135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40113
I20260925 19:00:24.296396 7135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
I20260925 19:00:24.298426 7304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40113 every 8 connection(s)
I20260925 19:00:24.299152 7305 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:00:24.299741 7305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: Bootstrap starting.
I20260925 19:00:24.302641 7305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:00:24.302860 7305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: Bootstrap complete.
I20260925 19:00:24.303033 7305 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } }
I20260925 19:00:24.303115 7305 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc5658ea063c468eb33f350dfa9ded66, State: Initialized, Role: LEARNER
I20260925 19:00:24.303148 7305 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } }
I20260925 19:00:24.303320 7305 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:24.303329 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } } }
I20260925 19:00:24.303545 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66 [sys.catalog]: This master's current role is: LEARNER
I20260925 19:00:24.304728 7319 catalog_manager.cc:1259] Loaded cluster ID: 80fea2c92a1a4e88b2ef5600d515a0be
I20260925 19:00:24.304806 7319 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: loading cluster ID for follower catalog manager: success
I20260925 19:00:24.305951 7319 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: acquiring CA information for follower catalog manager: success
I20260925 19:00:24.306228 7319 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc5658ea063c468eb33f350dfa9ded66: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 19:00:24.329378 7233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:24.329594 7233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:24.329630 7233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:24.330808 7233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:24.330871 7233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:24.330915 7233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:24.332322 7233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38317
--tserver_master_addrs=127.0.98.62:41805,127.0.98.61:39997,127.0.98.60:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:24.332592 7233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:24.332805 7233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:24.337522 7327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:24.337558 7324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:24.337538 7325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:24.337685 7233 server_base.cc:1034] running on GCE node
I20260925 19:00:24.337934 7233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:24.338172 7233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:24.339323 7233 hybrid_clock.cc:648] HybridClock initialized: now 1790362824339316 us; error 36 us; skew 500 ppm
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:24.341674 7233 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 19:00:24.342064 7233 webserver.cc:460] Webserver started at http://127.0.98.1:38317/ using document root <none> and password file <none>
I20260925 19:00:24.342321 7233 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:24.342403 7233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:24.343672 7233 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:00:24.344165 7334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.344316 7233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:24.344434 7233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "08b3a7016f664931adc2f06fd7aa1b28"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "5b97f5fbdc2bbb4191866dbb61d87679"
server_key_iv: "aee3145c9d7504891a962fce776969d5"
server_key_version: "encryptionkey@0"
I20260925 19:00:24.344751 7233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:24.389397 7233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:24.391404 7233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:24.391561 7233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:24.391835 7233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:24.392146 7233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:24.392195 7233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.392227 7233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:24.392259 7233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.398016 7233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39433
I20260925 19:00:24.398095 7447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39433 every 8 connection(s)
I20260925 19:00:24.398454 7233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 19:00:24.407685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 7233
I20260925 19:00:24.407857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 6680
I20260925 19:00:24.409929 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.399372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36295 (local address 127.0.98.62:41805)
0925 19:00:24.399532 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.399534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.399543 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:24.399973 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.399974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.399986 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.400035 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.400954 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.401449 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.403121 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.403390 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.403487 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.405991 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.405992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.405992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.405998 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.407845 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.408873 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.408874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.408875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.408930 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.409197 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.409198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.409199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.409348 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.409413 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.409769 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.409855 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260925 19:00:24.410414 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.399123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:36295)
0925 19:00:24.399357 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.399368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.399869 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.400118 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.400122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.400704 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.400860 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.400864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.401543 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.401544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.403027 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.403029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.403497 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.403498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.403631 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.404279 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.404290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.405905 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.408563 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.408568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.408570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.408774 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.409025 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.409025 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.409026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.409134 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.409439 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.409446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.409602 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.410025 (+ 423us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.410189 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":89,"thread_start_us":79,"threads_started":1}
I20260925 19:00:24.410509 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.399218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33091 (local address 127.0.98.61:39997)
0925 19:00:24.399274 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.399276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.399416 (+ 140us) server_negotiation.cc:407] Connection header received
0925 19:00:24.400106 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.400108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.400118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.400195 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.400967 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.401726 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.403185 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.403500 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.403540 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.407095 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.407096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.407097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.407106 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.409215 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.409569 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.409570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.409570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.409627 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.409817 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.409818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.409818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.409973 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.410034 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.410287 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.410370 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:00:24.410938 7449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41805
I20260925 19:00:24.411075 7449 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:24.411077 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.399123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39997 (local address 127.0.98.1:33091)
0925 19:00:24.399288 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.399334 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.399879 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.400208 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.400209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.400649 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.400862 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.400867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.401829 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.401830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.403082 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.403084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.403631 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.403632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.403685 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.404421 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.404425 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.407017 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.409286 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.409287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.409287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.409496 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.409689 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.409689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.409690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.409749 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.410104 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.410105 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.410150 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.410739 (+ 589us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.410864 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":76,"thread_start_us":81,"threads_started":1}
I20260925 19:00:24.411273 7449 heartbeater.cc:511] Master 127.0.98.62:41805 requested a full tablet report, sending...
I20260925 19:00:24.411680 6483 ts_manager.cc:194] Re-registered known tserver with Master: 08b3a7016f664931adc2f06fd7aa1b28 (127.0.98.1:39433)
I20260925 19:00:24.411845 7450 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39997
I20260925 19:00:24.411917 7450 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:24.411996 7450 heartbeater.cc:511] Master 127.0.98.61:39997 requested a full tablet report, sending...
I20260925 19:00:24.412155 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36295
I20260925 19:00:24.412348 7081 ts_manager.cc:194] Registered new tserver with Master: 08b3a7016f664931adc2f06fd7aa1b28 (127.0.98.1:39433)
I20260925 19:00:24.415009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45917
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41805
--webserver_port=41995
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41805,127.0.98.61:39997
--tserver_master_addrs=127.0.98.62:41805,127.0.98.61:39997,127.0.98.60:40113 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:00:24.416491 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.399428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32807 (local address 127.0.98.60:40113)
0925 19:00:24.399595 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.399597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.399707 (+ 110us) server_negotiation.cc:407] Connection header received
0925 19:00:24.400273 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.400274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.400288 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.400377 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.400967 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.401945 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.403190 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.403496 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.403555 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.406498 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.406499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.406500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.406507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.415336 (+ 8829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.415737 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.415738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.415738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.415787 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.415963 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.415963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.415964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.416055 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.416119 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.416276 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.416371 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":58,"threads_started":1}
I20260925 19:00:24.416594 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.399123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40113 (local address 127.0.98.1:32807)
0925 19:00:24.399643 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.399650 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.399882 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.400389 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.400389 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.400648 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.400862 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.400867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.402062 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.402062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.403082 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.403084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.403603 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.403604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.403665 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.404313 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.404317 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.406409 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.415429 (+ 9020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.415430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.415430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.415675 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.415838 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.415839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.415839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.415906 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.416111 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.416112 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.416155 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.416237 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.416350 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":307,"thread_start_us":64,"threads_started":1}
I20260925 19:00:24.417003 7448 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40113
I20260925 19:00:24.417061 7448 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:24.417130 7448 heartbeater.cc:511] Master 127.0.98.60:40113 requested a full tablet report, sending...
I20260925 19:00:24.417487 7269 ts_manager.cc:194] Registered new tserver with Master: 08b3a7016f664931adc2f06fd7aa1b28 (127.0.98.1:39433)
W20260925 19:00:24.494504 7456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:24.494683 7456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:24.494705 7456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:24.495750 7456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:24.495801 7456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:24.495826 7456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:24.497159 7456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41995
--tserver_master_addrs=127.0.98.62:41805,127.0.98.61:39997,127.0.98.60:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:24.497327 7456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:24.497498 7456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:00:24.502524 7456 server_base.cc:1034] running on GCE node
W20260925 19:00:24.502498 7464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:24.502523 7462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:24.502489 7461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:24.503053 7456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:24.503331 7456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:24.504494 7456 hybrid_clock.cc:648] HybridClock initialized: now 1790362824504488 us; error 31 us; skew 500 ppm
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:24.506903 7456 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 19:00:24.507289 7456 webserver.cc:460] Webserver started at http://127.0.98.2:41995/ using document root <none> and password file <none>
I20260925 19:00:24.507537 7456 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:24.507607 7456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:24.508971 7456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:00:24.509518 7471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.509698 7456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:24.509778 7456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "3b3357daaf754dee986c0af6cbdc6f1f"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "09d63029d7c6dc3a31816330f2eee84d"
server_key_iv: "9022b16ef1d43eced045536dd8058a77"
server_key_version: "encryptionkey@0"
I20260925 19:00:24.510105 7456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:24.523078 7456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:24.525015 7456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:24.525146 7456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:24.525365 7456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:24.525637 7456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:24.525673 7456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.525703 7456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:24.525727 7456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.531231 7456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45917
I20260925 19:00:24.531271 7584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45917 every 8 connection(s)
I20260925 19:00:24.531553 7456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 19:00:24.539754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 7456
I20260925 19:00:24.539911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 6814
I20260925 19:00:24.540872 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.532257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45129 (local address 127.0.98.62:41805)
0925 19:00:24.532340 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.532342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.532595 (+ 253us) server_negotiation.cc:407] Connection header received
0925 19:00:24.533070 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.533071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.533083 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.533113 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.533806 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.534513 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.535573 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.535807 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.535871 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.537755 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.537756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.537757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.537762 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.539460 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.539939 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.539941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.539941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.539999 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.540246 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.540247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.540247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.540373 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.540427 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.540710 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.540786 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:00:24.541227 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.532135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:45129)
0925 19:00:24.532444 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.532481 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.532957 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.533535 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.533536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.533674 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.533720 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.533721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.534602 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.534603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.535474 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.535476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.535871 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.535872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.535930 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.536325 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.536330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.537676 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.539543 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.539549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.539551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.539838 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.540073 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.540074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.540074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.540173 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.540452 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.540458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.540580 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.540678 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.540825 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":40,"thread_start_us":176,"threads_started":1}
I20260925 19:00:24.541738 7586 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41805
I20260925 19:00:24.541831 7586 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:24.541958 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.532307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44245 (local address 127.0.98.61:39997)
0925 19:00:24.532379 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.532380 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.532571 (+ 191us) server_negotiation.cc:407] Connection header received
0925 19:00:24.533070 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.533071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.533083 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.533113 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.533805 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.534520 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.535591 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.535863 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.535929 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.538792 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.538794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.538794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.538802 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.540731 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.541140 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.541141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.541141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.541188 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.541365 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.541365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.541365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.541510 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.541549 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.541786 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.541880 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:00:24.541990 7586 heartbeater.cc:511] Master 127.0.98.62:41805 requested a full tablet report, sending...
I20260925 19:00:24.542011 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.532135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39997 (local address 127.0.98.2:44245)
0925 19:00:24.532444 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.532481 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.532957 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.533178 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.533181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.533576 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.533705 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.533711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.534602 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.534603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.535485 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.535486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.535978 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.535979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.536042 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.536470 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.536474 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.538712 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.540830 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.540831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.540832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.541070 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.541250 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.541251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.541251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.541303 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.541601 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.541602 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.541659 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.541748 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.541865 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":58,"thread_start_us":94,"threads_started":1}
I20260925 19:00:24.542374 6483 ts_manager.cc:194] Re-registered known tserver with Master: 3b3357daaf754dee986c0af6cbdc6f1f (127.0.98.2:45917)
I20260925 19:00:24.542711 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45129
I20260925 19:00:24.545208 7587 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39997
I20260925 19:00:24.545261 7587 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:24.545337 7587 heartbeater.cc:511] Master 127.0.98.61:39997 requested a full tablet report, sending...
I20260925 19:00:24.545625 7081 ts_manager.cc:194] Registered new tserver with Master: 3b3357daaf754dee986c0af6cbdc6f1f (127.0.98.2:45917)
I20260925 19:00:24.546566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37301
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41805
--webserver_port=45075
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41805,127.0.98.61:39997
--tserver_master_addrs=127.0.98.62:41805,127.0.98.61:39997,127.0.98.60:40113 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:00:24.547480 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.532262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59371 (local address 127.0.98.60:40113)
0925 19:00:24.532360 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.532362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.532641 (+ 279us) server_negotiation.cc:407] Connection header received
0925 19:00:24.533040 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.533041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.533056 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.533117 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.533788 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.534384 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.535342 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.535600 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.535657 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.538151 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.538152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.538152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.538159 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.539935 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.546745 (+ 6810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.546746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.546747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.546804 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.546990 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.546991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.546991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.547096 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.547164 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.547320 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.547398 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 19:00:24.547493 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.532136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40113 (local address 127.0.98.2:59371)
0925 19:00:24.532584 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.532594 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.532964 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.533178 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.533181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.533538 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.533705 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.533708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.534473 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.534473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.535232 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.535234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.535668 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.535669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.535790 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.536197 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.536206 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.538081 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.546396 (+ 8315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.546396 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.546397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.546546 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.546867 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.546868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.546869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.546943 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.547156 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.547156 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.547203 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.547286 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.547397 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":64,"thread_start_us":219,"threads_started":1}
I20260925 19:00:24.548089 7585 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40113
I20260925 19:00:24.548141 7585 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:24.548189 7585 heartbeater.cc:511] Master 127.0.98.60:40113 requested a full tablet report, sending...
I20260925 19:00:24.548480 7269 ts_manager.cc:194] Registered new tserver with Master: 3b3357daaf754dee986c0af6cbdc6f1f (127.0.98.2:45917)
W20260925 19:00:24.623373 7592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:24.623550 7592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:24.623570 7592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:24.624562 7592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:24.624598 7592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:24.624621 7592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:24.625991 7592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45075
--tserver_master_addrs=127.0.98.62:41805,127.0.98.61:39997,127.0.98.60:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:24.626200 7592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:24.626441 7592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:24.630844 7600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:24.630831 7598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:24.630949 7592 server_base.cc:1034] running on GCE node
W20260925 19:00:24.630828 7597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:24.631217 7592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:24.631402 7592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:24.632529 7592 hybrid_clock.cc:648] HybridClock initialized: now 1790362824632521 us; error 24 us; skew 500 ppm
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:24.634809 7592 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 19:00:24.635202 7592 webserver.cc:460] Webserver started at http://127.0.98.3:45075/ using document root <none> and password file <none>
I20260925 19:00:24.635422 7592 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:24.635493 7592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:24.636691 7592 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:24.637205 7607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.637362 7592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.637452 7592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "3a933feeabe44a0db2f6dda09fb121b9"
format_stamp: "Formatted at 2026-09-25 19:00:23 on dist-test-slave-49bz"
server_key: "9d7eab26aa76f386d50e25c5fcaae4e1"
server_key_iv: "d9950716b54e2017db66bcf0507cdaa0"
server_key_version: "encryptionkey@0"
I20260925 19:00:24.637756 7592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:24.652344 7592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:24.653916 7592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:24.654047 7592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:24.654234 7592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:24.654579 7592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:24.654625 7592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.654661 7592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:24.654685 7592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:24.659960 7592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37301
I20260925 19:00:24.659996 7720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37301 every 8 connection(s)
I20260925 19:00:24.660319 7592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:00:24.660737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 7592
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:24.671439 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.662183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.98.60:40113)
0925 19:00:24.662331 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.662342 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.662973 (+ 631us) server_negotiation.cc:407] Connection header received
0925 19:00:24.662995 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.662996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.663009 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.663063 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.663903 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.664648 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.665569 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.665861 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.665897 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.667980 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.667982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.667982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.667989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.669736 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.670144 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.670145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.670146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.670213 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.670435 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.670436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.670436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.670574 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.670664 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.670850 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.671313 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260925 19:00:24.673666 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.661519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42765 (local address 127.0.98.61:39997)
0925 19:00:24.661564 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.661566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.662734 (+ 1168us) server_negotiation.cc:407] Connection header received
0925 19:00:24.662912 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.662914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.662925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.662956 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.663709 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.664428 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.665828 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.666124 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.666161 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.670332 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.670333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.670333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.670340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.672421 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.672857 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.672858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.672858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.672903 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.673112 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.673113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.673114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.673239 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.673282 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.673490 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.673578 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:00:24.673695 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.661004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:48841)
0925 19:00:24.661179 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.661216 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.662501 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.662820 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.662824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.663355 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.663605 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.663610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.664535 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.664536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.665717 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.665719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.666225 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.666225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.666386 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.667437 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.667441 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.669354 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.671966 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.671978 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.671980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.672344 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.672585 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.672586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.672587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.672719 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.673016 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.673022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.673176 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.673285 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.673468 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":31,"thread_start_us":102,"threads_started":1}
I20260925 19:00:24.673995 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.661325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39997 (local address 127.0.98.3:42765)
0925 19:00:24.662665 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.662674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.662824 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.663020 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.663021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.663368 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.663605 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.663610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.664536 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.664536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.665717 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.665719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.666227 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.666227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.666386 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.667016 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.667025 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.670199 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.672525 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.672526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.672526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.672748 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.672958 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.672959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.672959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.673032 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.673333 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.673333 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.673390 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.673772 (+ 382us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.673874 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"mutex_wait_us":75,"thread_start_us":67,"threads_started":1}
I20260925 19:00:24.674086 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.661738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48841 (local address 127.0.98.62:41805)
0925 19:00:24.661803 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.661805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.661815 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:00:24.662630 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.662631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.662642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.662687 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.663706 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.664422 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.665822 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.666124 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.666173 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.669449 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.669450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.669450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.669457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.671590 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.672461 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.672462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.672462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.672518 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.672807 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.672808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.672808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.672931 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.672989 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.673924 (+ 935us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.673989 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 19:00:24.674327 7723 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39997
I20260925 19:00:24.674424 7723 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:24.674597 7723 heartbeater.cc:511] Master 127.0.98.61:39997 requested a full tablet report, sending...
I20260925 19:00:24.674657 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.662573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.0.98.62:41805)
0925 19:00:24.663584 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.663586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.663736 (+ 150us) server_negotiation.cc:407] Connection header received
0925 19:00:24.663758 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.663759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.663771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.663841 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.664505 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.665160 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.666057 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.666405 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.666462 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.669224 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.669226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.669226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.669233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.673493 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.673826 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.673827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.673827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.673879 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.674068 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.674069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.674069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.674190 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.674286 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.674471 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.674539 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":76,"threads_started":1}
I20260925 19:00:24.674723 7722 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41805
I20260925 19:00:24.674758 7722 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:24.674815 7722 heartbeater.cc:511] Master 127.0.98.62:41805 requested a full tablet report, sending...
I20260925 19:00:24.675002 7081 ts_manager.cc:194] Registered new tserver with Master: 3a933feeabe44a0db2f6dda09fb121b9 (127.0.98.3:37301)
I20260925 19:00:24.675308 6483 ts_manager.cc:194] Re-registered known tserver with Master: 3a933feeabe44a0db2f6dda09fb121b9 (127.0.98.3:37301)
Sep 25 19:00:24 dist-test-slave-49bz krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 19:00:24.675809 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48841
I20260925 19:00:24.676790 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.661934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.0.98.61:39997)
0925 19:00:24.663332 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.663399 (+ 67us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.663408 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:24.663429 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.663430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.663441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.663508 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.667275 (+ 3767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.667927 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.670606 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.670984 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.671066 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.672561 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.672562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.672563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.672569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.675235 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.676080 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.676081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.676081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.676135 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.676322 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.676323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.676323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.676430 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.676499 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.676666 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.676737 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1356,"thread_start_us":67,"threads_started":1}
I20260925 19:00:24.676190 6483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:00:24.678608 6483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:24.679004 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.661803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53917 (local address 127.0.98.60:40113)
0925 19:00:24.661852 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.661854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.661864 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:00:24.662611 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.662612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.662625 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.662678 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.666657 (+ 3979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.667341 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.669849 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.670193 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.670232 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.676208 (+ 5976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.676209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.676210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.676226 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.677706 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.678061 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.678062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.678062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.678109 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.678340 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.678341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.678341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.678442 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.678512 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.678859 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.678925 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:00:24.679140 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.661004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40113 (local address 127.0.98.3:53917)
0925 19:00:24.661248 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.661259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.662484 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.662820 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.662824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.663351 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:24.666600 (+ 3249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.666602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.667991 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.667991 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.668758 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.668761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.670351 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.670352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.670428 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.671605 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:24.671609 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:24.676124 (+ 4515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:24.677797 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.677798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.677799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.677985 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.678193 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:24.678194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:24.678195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:24.678285 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:24.678702 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:24.678703 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:24.678743 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.678816 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.678916 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260925 19:00:24.679477 7721 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40113
I20260925 19:00:24.679522 7721 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:24.679575 7721 heartbeater.cc:511] Master 127.0.98.60:40113 requested a full tablet report, sending...
I20260925 19:00:24.679837 7269 ts_manager.cc:194] Registered new tserver with Master: 3a933feeabe44a0db2f6dda09fb121b9 (127.0.98.3:37301)
I20260925 19:00:24.685617 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.682275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.0.98.3:37301)
0925 19:00:24.682577 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.682582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.683106 (+ 524us) server_negotiation.cc:407] Connection header received
0925 19:00:24.683174 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.683176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.683187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.683252 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:24.683330 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.683890 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.684824 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.685294 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.685385 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.685448 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.685522 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.685553 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":87,"threads_started":1}
I20260925 19:00:24.685618 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.682165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37301 (local address 127.0.0.1:33046)
0925 19:00:24.683043 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.683050 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.683109 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.683250 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.683251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.683252 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:24.683286 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.683288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.683965 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.683966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.684728 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.684730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.685386 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.685387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.685448 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.685453 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.685510 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.685552 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":483,"thread_start_us":70,"threads_started":1}
I20260925 19:00:24.686427 7655 tablet_service.cc:1467] Processing CreateTablet for tablet 7e26e9d6f9894efa830fec05777e0481 (DEFAULT_TABLE table=test-table [id=1443cd616ee3449b8dc12a68c873f502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:24.686745 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e26e9d6f9894efa830fec05777e0481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:24.686955 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.682164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45917 (local address 127.0.0.1:40030)
0925 19:00:24.682425 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.682436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.682484 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.682739 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.682740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.682742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:24.682781 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.682783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.684865 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.684865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.685884 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.685886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.686737 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.686738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.686795 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.686800 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.686853 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.686888 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260925 19:00:24.686969 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.682399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.0.98.2:45917)
0925 19:00:24.682591 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.682595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.682603 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:24.682638 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.682639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.682654 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.682742 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:24.682844 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.684761 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.685980 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.686680 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.686737 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.686804 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.686866 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.686901 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
I20260925 19:00:24.687137 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.682289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39433 (local address 127.0.0.1:50256)
0925 19:00:24.682554 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.682565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.682601 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.683006 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.683007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.683009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:24.683046 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.683048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.684631 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.684632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.685669 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.685671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.686928 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.686929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.686984 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.686987 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.687037 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.687078 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":76,"threads_started":1}
I20260925 19:00:24.687235 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.682403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.0.98.1:39433)
0925 19:00:24.682860 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.682864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.682872 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:24.682904 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.682906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.682922 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.682980 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:24.683156 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.684553 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.685754 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.686503 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.686586 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.686648 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.687117 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.687157 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":73,"threads_started":1}
I20260925 19:00:24.687508 7519 tablet_service.cc:1467] Processing CreateTablet for tablet 7e26e9d6f9894efa830fec05777e0481 (DEFAULT_TABLE table=test-table [id=1443cd616ee3449b8dc12a68c873f502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:24.687796 7519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e26e9d6f9894efa830fec05777e0481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:24.688136 7382 tablet_service.cc:1467] Processing CreateTablet for tablet 7e26e9d6f9894efa830fec05777e0481 (DEFAULT_TABLE table=test-table [id=1443cd616ee3449b8dc12a68c873f502]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:24.688422 7382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e26e9d6f9894efa830fec05777e0481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:24.689957 7747 tablet_bootstrap.cc:492] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: Bootstrap starting.
I20260925 19:00:24.690137 7748 tablet_bootstrap.cc:492] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: Bootstrap starting.
I20260925 19:00:24.690665 7747 tablet_bootstrap.cc:654] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:24.690665 7748 tablet_bootstrap.cc:654] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:24.690982 7747 log.cc:824] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:24.690982 7748 log.cc:824] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:24.691771 7747 tablet_bootstrap.cc:492] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: No bootstrap required, opened a new log
I20260925 19:00:24.691829 7748 tablet_bootstrap.cc:492] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: No bootstrap required, opened a new log
I20260925 19:00:24.691859 7747 ts_tablet_manager.cc:1397] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:00:24.691903 7748 ts_tablet_manager.cc:1397] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:24.691919 7751 tablet_bootstrap.cc:492] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: Bootstrap starting.
I20260925 19:00:24.692988 7751 tablet_bootstrap.cc:654] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:24.693269 7751 log.cc:824] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:24.693391 7748 raft_consensus.cc:348] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 } } peers { permanent_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45917 } } peers { permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 } }
I20260925 19:00:24.693495 7748 raft_consensus.cc:374] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:24.693512 7748 raft_consensus.cc:729] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3357daaf754dee986c0af6cbdc6f1f, State: Initialized, Role: FOLLOWER
I20260925 19:00:24.693610 7748 consensus_queue.cc:260] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 } } peers { permanent_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45917 } } peers { permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 } }
I20260925 19:00:24.693799 7748 ts_tablet_manager.cc:1428] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:00:24.693941 7586 heartbeater.cc:503] Master 127.0.98.62:41805 was elected leader, sending a full tablet report...
I20260925 19:00:24.694800 7751 tablet_bootstrap.cc:492] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: No bootstrap required, opened a new log
I20260925 19:00:24.694872 7751 ts_tablet_manager.cc:1397] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:00:24.695138 7747 raft_consensus.cc:348] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 } } peers { permanent_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45917 } } peers { permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 } }
I20260925 19:00:24.695250 7747 raft_consensus.cc:374] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:24.695281 7747 raft_consensus.cc:729] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a933feeabe44a0db2f6dda09fb121b9, State: Initialized, Role: FOLLOWER
I20260925 19:00:24.695395 7747 consensus_queue.cc:260] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 } } peers { permanent_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45917 } } peers { permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 } }
I20260925 19:00:24.695640 7747 ts_tablet_manager.cc:1428] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260925 19:00:24.695926 7722 heartbeater.cc:503] Master 127.0.98.62:41805 was elected leader, sending a full tablet report...
I20260925 19:00:24.696552 7751 raft_consensus.cc:348] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 } } peers { permanent_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45917 } } peers { permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 } }
I20260925 19:00:24.696653 7751 raft_consensus.cc:374] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:24.696672 7751 raft_consensus.cc:729] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b3a7016f664931adc2f06fd7aa1b28, State: Initialized, Role: FOLLOWER
I20260925 19:00:24.696733 7751 consensus_queue.cc:260] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 } } peers { permanent_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45917 } } peers { permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 } }
I20260925 19:00:24.696894 7751 ts_tablet_manager.cc:1428] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:00:24.696938 7449 heartbeater.cc:503] Master 127.0.98.62:41805 was elected leader, sending a full tablet report...
I20260925 19:00:24.700305 7753 raft_consensus.cc:482] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:24.700407 7753 raft_consensus.cc:504] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 } } peers { permanent_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45917 } } peers { permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 } }
I20260925 19:00:24.700703 7753 leader_election.cc:288] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a933feeabe44a0db2f6dda09fb121b9 (127.0.98.3:37301), 08b3a7016f664931adc2f06fd7aa1b28 (127.0.98.1:39433)
I20260925 19:00:24.704151 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.700844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53515 (local address 127.0.98.3:37301)
0925 19:00:24.700885 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.700888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.700897 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:24.700987 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.700988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.701000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.701060 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:24.701212 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.701917 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.703303 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.703837 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.703904 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.703911 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.704049 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.704080 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:00:24.704151 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.700945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49739 (local address 127.0.98.1:39433)
0925 19:00:24.700978 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.700980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.700986 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:00:24.701004 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.701005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.701013 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.701064 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:24.701212 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.701895 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.703287 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.703837 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.703904 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.703911 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.704049 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.704079 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:00:24.704169 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.700808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37301 (local address 127.0.98.2:53515)
0925 19:00:24.700851 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.700861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.700897 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.701065 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.701065 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.701067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:24.701148 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.701149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.701986 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.701987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.703200 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.703203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.703905 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.703906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.703987 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.703989 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.704036 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.704076 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":76}
I20260925 19:00:24.704182 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.700862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39433 (local address 127.0.98.2:49739)
0925 19:00:24.700893 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:24.700902 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:24.700934 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:24.701082 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:24.701083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:24.701084 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:24.701148 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.701149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.701986 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.701987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.703200 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:24.703203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.703905 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.703906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:24.703977 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.703982 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:00:24.704036 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 19:00:24.704076 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":40}
I20260925 19:00:24.704748 7675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e26e9d6f9894efa830fec05777e0481" candidate_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a933feeabe44a0db2f6dda09fb121b9" is_pre_election: true
I20260925 19:00:24.704785 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e26e9d6f9894efa830fec05777e0481" candidate_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b3a7016f664931adc2f06fd7aa1b28" is_pre_election: true
I20260925 19:00:24.704882 7402 raft_consensus.cc:2454] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b3357daaf754dee986c0af6cbdc6f1f in term 0.
I20260925 19:00:24.704882 7675 raft_consensus.cc:2454] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b3357daaf754dee986c0af6cbdc6f1f in term 0.
I20260925 19:00:24.705081 7475 leader_election.cc:302] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a933feeabe44a0db2f6dda09fb121b9, 3b3357daaf754dee986c0af6cbdc6f1f; no voters:
I20260925 19:00:24.705242 7753 raft_consensus.cc:2781] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:24.705276 7753 raft_consensus.cc:482] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:24.705291 7753 raft_consensus.cc:3037] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:24.705829 7753 raft_consensus.cc:504] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 } } peers { permanent_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45917 } } peers { permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 } }
I20260925 19:00:24.705963 7753 leader_election.cc:288] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [CANDIDATE]: Term 1 election: Requested vote from peers 3a933feeabe44a0db2f6dda09fb121b9 (127.0.98.3:37301), 08b3a7016f664931adc2f06fd7aa1b28 (127.0.98.1:39433)
I20260925 19:00:24.706074 7675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e26e9d6f9894efa830fec05777e0481" candidate_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a933feeabe44a0db2f6dda09fb121b9"
I20260925 19:00:24.706085 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e26e9d6f9894efa830fec05777e0481" candidate_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b3a7016f664931adc2f06fd7aa1b28"
I20260925 19:00:24.706157 7675 raft_consensus.cc:3037] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:24.706156 7402 raft_consensus.cc:3037] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:24.706809 7675 raft_consensus.cc:2454] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b3357daaf754dee986c0af6cbdc6f1f in term 1.
I20260925 19:00:24.706816 7402 raft_consensus.cc:2454] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b3357daaf754dee986c0af6cbdc6f1f in term 1.
I20260925 19:00:24.706998 7473 leader_election.cc:302] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b3a7016f664931adc2f06fd7aa1b28, 3b3357daaf754dee986c0af6cbdc6f1f; no voters:
I20260925 19:00:24.707118 7753 raft_consensus.cc:2781] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:24.707269 7753 raft_consensus.cc:686] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 1 LEADER]: Becoming Leader. State: Replica: 3b3357daaf754dee986c0af6cbdc6f1f, State: Running, Role: LEADER
I20260925 19:00:24.707363 7753 consensus_queue.cc:237] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 } } peers { permanent_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45917 } } peers { permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 } }
I20260925 19:00:24.707988 6483 catalog_manager.cc:5495] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b3357daaf754dee986c0af6cbdc6f1f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b3357daaf754dee986c0af6cbdc6f1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 } health_report { overall_health: UNKNOWN } } }
I20260925 19:00:24.714429 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.712557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.0.98.2:45917)
0925 19:00:24.712611 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.712613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.712620 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:00:24.712667 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.712668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.712680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.712729 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:24.712823 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.713207 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.713850 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.714002 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.714046 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.714152 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:24.714222 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:24.714319 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.714336 (+ 17us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.714350 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:00:24.715663 7402 raft_consensus.cc:1261] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28 [term 1 FOLLOWER]: Refusing update from remote peer 3b3357daaf754dee986c0af6cbdc6f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:24.715682 7675 raft_consensus.cc:1261] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b3357daaf754dee986c0af6cbdc6f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:24.715921 7753 consensus_queue.cc:1035] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a933feeabe44a0db2f6dda09fb121b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:24.716037 7756 consensus_queue.cc:1035] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b3a7016f664931adc2f06fd7aa1b28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:24.718467 7759 mvcc.cc:205] Tried to move back new op lower bound from 7333326130033479680 to 7333326130001960960. Current Snapshot: MvccSnapshot[applied={T|T < 7333326130033479680}]
I20260925 19:00:24.718688 7760 mvcc.cc:205] Tried to move back new op lower bound from 7333326130033479680 to 7333326130001960960. Current Snapshot: MvccSnapshot[applied={T|T < 7333326130033479680}]
I20260925 19:00:24.718993 7761 mvcc.cc:205] Tried to move back new op lower bound from 7333326130033479680 to 7333326130001960960. Current Snapshot: MvccSnapshot[applied={T|T < 7333326130033479680}]
I20260925 19:00:24.721354 6484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:24.721446 6484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:24.722482 6484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5986923f01141dfa7d222038a24e3c1: Sending DeleteTablet for 3 replicas of tablet 7e26e9d6f9894efa830fec05777e0481
I20260925 19:00:24.722750 7655 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e26e9d6f9894efa830fec05777e0481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33046
I20260925 19:00:24.722816 7519 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e26e9d6f9894efa830fec05777e0481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40030
I20260925 19:00:24.722988 7769 tablet_replica.cc:330] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: stopping tablet replica
I20260925 19:00:24.722998 7770 tablet_replica.cc:330] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: stopping tablet replica
I20260925 19:00:24.723004 7382 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e26e9d6f9894efa830fec05777e0481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50256
I20260925 19:00:24.723075 7770 raft_consensus.cc:2229] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:24.723181 7769 raft_consensus.cc:2229] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:24.723218 7770 raft_consensus.cc:2258] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:24.723343 7769 raft_consensus.cc:2258] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:24.723745 7769 ts_tablet_manager.cc:1905] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:24.724044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 6452
I20260925 19:00:24.725073 7770 ts_tablet_manager.cc:1905] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:24.726193 7769 ts_tablet_manager.cc:1918] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:24.726284 7769 log.cc:1196] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/7e26e9d6f9894efa830fec05777e0481
I20260925 19:00:24.726483 7770 ts_tablet_manager.cc:1918] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:24.726539 7769 ts_tablet_manager.cc:1939] T 7e26e9d6f9894efa830fec05777e0481 P 3a933feeabe44a0db2f6dda09fb121b9: Deleting consensus metadata
I20260925 19:00:24.726547 7770 log.cc:1196] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/7e26e9d6f9894efa830fec05777e0481
I20260925 19:00:24.726783 7770 ts_tablet_manager.cc:1939] T 7e26e9d6f9894efa830fec05777e0481 P 3b3357daaf754dee986c0af6cbdc6f1f: Deleting consensus metadata
W20260925 19:00:24.729738 7472 connection.cc:535] server connection from 127.0.0.1:40030 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 19:00:24.729995 7608 connection.cc:535] server connection from 127.0.0.1:33046 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:00:24.734026 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.723986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.0.98.60:40113)
0925 19:00:24.724024 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.724026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.724033 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:00:24.724052 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.724053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.724062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.724114 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.727746 (+ 3632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.728447 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.729448 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.729726 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.729767 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.731400 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.731402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.731402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.731408 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.732937 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.733284 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.733285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.733286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.733332 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.733523 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.733524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.733524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.733631 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.733697 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.733874 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.733940 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:24.735219 7771 tablet_replica.cc:330] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: stopping tablet replica
I20260925 19:00:24.735322 7771 raft_consensus.cc:2229] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:24.735433 7771 raft_consensus.cc:2258] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:24.735877 7771 ts_tablet_manager.cc:1905] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:24.737267 7771 ts_tablet_manager.cc:1918] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:24.737339 7771 log.cc:1196] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/7e26e9d6f9894efa830fec05777e0481
I20260925 19:00:24.737593 7771 ts_tablet_manager.cc:1939] T 7e26e9d6f9894efa830fec05777e0481 P 08b3a7016f664931adc2f06fd7aa1b28: Deleting consensus metadata
W20260925 19:00:24.737872 7335 connection.cc:441] server connection from 127.0.0.1:50256 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50256 (request call id 1) could send its response
I20260925 19:00:24.738965 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.731676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.0.98.60:40113)
0925 19:00:24.731708 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.731710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.731845 (+ 135us) server_negotiation.cc:407] Connection header received
0925 19:00:24.731932 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.731933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.731945 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.731979 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.732551 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.733200 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.734056 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.734422 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.734460 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.736095 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.736096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.736097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.736103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.737704 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.738148 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.738148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.738149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.738203 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.738465 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.738465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.738466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.738590 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.738680 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.738801 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.738875 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:00:24.739748 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:24.731515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.0.98.61:39997)
0925 19:00:24.731553 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:24.731555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:24.731599 (+ 44us) server_negotiation.cc:407] Connection header received
0925 19:00:24.731684 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:24.731685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:24.731697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:24.731726 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:24.732381 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.733050 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.734047 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:24.734641 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:24.734686 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:24.736391 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:24.736393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:24.736393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:24.736399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:24.738455 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.738788 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.738789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.738789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.738840 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:24.739246 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:24.739247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:24.739247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:24.739406 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:24.739486 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:24.739583 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 19:00:24.739657 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:00:25.719460 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:25.719183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:51991)
0925 19:00:25.719329 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:25.719400 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260925 19:00:25.719897 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:25.719573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:48161)
0925 19:00:25.719767 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:25.719846 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":48,"threads_started":1}
W20260925 19:00:25.719980 7586 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41805 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
I20260925 19:00:25.720134 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:25.719827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:57317)
0925 19:00:25.720019 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:25.720088 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":53,"threads_started":1}
W20260925 19:00:25.720311 7449 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41805 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
W20260925 19:00:25.720507 7722 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41805 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
I20260925 19:00:26.223304 7782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5986923f01141dfa7d222038a24e3c1)
I20260925 19:00:26.223489 7782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } }
I20260925 19:00:26.223731 7782 leader_election.cc:288] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805)
I20260925 19:00:26.224082 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:26.223903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:45782)
0925 19:00:26.223997 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:26.224047 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":51,"threads_started":1}
W20260925 19:00:26.224203 7067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
I20260925 19:00:26.224548 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:26.224455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:45784)
0925 19:00:26.224503 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:26.224530 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 19:00:26.224633 7067 leader_election.cc:334] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
I20260925 19:00:26.224704 7067 leader_election.cc:302] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3580c5192ad74c98bbeb4254f62d1fac; no voters: b5986923f01141dfa7d222038a24e3c1
I20260925 19:00:26.224809 7782 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:00:26.720808 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:26.720333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:47273)
0925 19:00:26.720709 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:26.720754 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":71,"threads_started":1}
I20260925 19:00:26.720850 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:26.720589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:35913)
0925 19:00:26.720754 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:26.720798 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
I20260925 19:00:26.721192 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:26.720875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:41177)
0925 19:00:26.721090 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:26.721127 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":52,"threads_started":1}
I20260925 19:00:27.721522 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:27.721236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:47985)
0925 19:00:27.721433 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:27.721474 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":107,"threads_started":1}
I20260925 19:00:27.721520 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:27.721236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:42647)
0925 19:00:27.721430 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:27.721464 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":104,"threads_started":1}
I20260925 19:00:27.721779 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:27.721582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:38609)
0925 19:00:27.721713 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:27.721744 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":50,"threads_started":1}
I20260925 19:00:28.298486 7791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5986923f01141dfa7d222038a24e3c1)
I20260925 19:00:28.298581 7791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } }
I20260925 19:00:28.298746 7791 leader_election.cc:288] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805)
I20260925 19:00:28.299048 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:28.298883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:43956)
0925 19:00:28.298986 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:28.299015 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":54,"threads_started":1}
I20260925 19:00:28.299434 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:28.299313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:43968)
0925 19:00:28.299375 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:28.299417 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 19:00:28.299508 7067 leader_election.cc:334] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
I20260925 19:00:28.299578 7067 leader_election.cc:302] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3580c5192ad74c98bbeb4254f62d1fac; no voters: b5986923f01141dfa7d222038a24e3c1
I20260925 19:00:28.299654 7791 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:00:28.722132 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:28.721897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:46165)
0925 19:00:28.722043 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:28.722081 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260925 19:00:28.722132 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:28.721942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:45809)
0925 19:00:28.722065 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:28.722094 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260925 19:00:28.722302 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:28.722107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:41833)
0925 19:00:28.722199 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:28.722234 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":53,"threads_started":1}
I20260925 19:00:29.722721 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:29.722493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:36463)
0925 19:00:29.722624 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:29.722659 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260925 19:00:29.722869 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:29.722668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:36903)
0925 19:00:29.722797 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:29.722826 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":50,"threads_started":1}
I20260925 19:00:29.722869 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:29.722659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:47161)
0925 19:00:29.722797 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:29.722826 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260925 19:00:30.143550 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5986923f01141dfa7d222038a24e3c1)
I20260925 19:00:30.143652 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } }
I20260925 19:00:30.143824 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805)
I20260925 19:00:30.144165 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:30.143983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:44174)
0925 19:00:30.144100 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:30.144130 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260925 19:00:30.144554 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:30.144448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:44180)
0925 19:00:30.144502 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:30.144534 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:00:30.144647 7067 leader_election.cc:334] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
I20260925 19:00:30.144698 7067 leader_election.cc:302] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3580c5192ad74c98bbeb4254f62d1fac; no voters: b5986923f01141dfa7d222038a24e3c1
I20260925 19:00:30.144764 7800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:00:30.723413 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:30.723204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:38471)
0925 19:00:30.723325 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:30.723364 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260925 19:00:30.723416 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:30.723204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:51229)
0925 19:00:30.723344 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:30.723375 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260925 19:00:30.723416 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:30.723198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:47017)
0925 19:00:30.723343 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:30.723375 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260925 19:00:31.723997 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:31.723787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:58451)
0925 19:00:31.723918 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:31.723953 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260925 19:00:31.724009 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:31.723787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:35643)
0925 19:00:31.723931 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:31.723968 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260925 19:00:31.724009 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:31.723839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:50877)
0925 19:00:31.723944 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:31.723967 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":41,"threads_started":1}
I20260925 19:00:31.880024 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5986923f01141dfa7d222038a24e3c1)
I20260925 19:00:31.880125 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } }
I20260925 19:00:31.880281 7809 leader_election.cc:288] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805)
I20260925 19:00:31.880604 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:31.880426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:44346)
0925 19:00:31.880539 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:31.880568 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
W20260925 19:00:31.880683 7067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260925 19:00:31.881011 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:31.880922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:44354)
0925 19:00:31.880967 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:31.880993 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:00:31.881093 7067 leader_election.cc:334] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
I20260925 19:00:31.881145 7067 leader_election.cc:302] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3580c5192ad74c98bbeb4254f62d1fac; no voters: b5986923f01141dfa7d222038a24e3c1
I20260925 19:00:31.881225 7809 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:00:32.724581 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:32.724339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:51205)
0925 19:00:32.724501 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:32.724533 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":55,"threads_started":1}
I20260925 19:00:32.724581 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:32.724378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:57509)
0925 19:00:32.724486 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:32.724521 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
I20260925 19:00:32.724624 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:32.724457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:47681)
0925 19:00:32.724562 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:32.724586 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":41,"threads_started":1}
I20260925 19:00:33.725222 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:33.724968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:34149)
0925 19:00:33.725128 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:33.725168 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260925 19:00:33.725224 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:33.724992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:59445)
0925 19:00:33.725124 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:33.725161 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260925 19:00:33.725306 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:33.725112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:43247)
0925 19:00:33.725220 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:33.725251 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260925 19:00:34.725795 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:34.725587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:46343)
0925 19:00:34.725708 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:34.725748 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260925 19:00:34.725795 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:34.725585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:58115)
0925 19:00:34.725708 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:34.725748 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260925 19:00:34.725934 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:34.725744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:38053)
0925 19:00:34.725837 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:34.725889 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":55,"threads_started":1}
I20260925 19:00:35.435370 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5986923f01141dfa7d222038a24e3c1)
I20260925 19:00:35.435458 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } }
I20260925 19:00:35.435591 7821 leader_election.cc:288] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805)
I20260925 19:00:35.435932 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:35.435728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:44618)
0925 19:00:35.435858 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:35.435894 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260925 19:00:35.436273 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:35.436195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:44634)
0925 19:00:35.436237 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:35.436259 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 19:00:35.436350 7067 leader_election.cc:334] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
I20260925 19:00:35.436411 7067 leader_election.cc:302] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3580c5192ad74c98bbeb4254f62d1fac; no voters: b5986923f01141dfa7d222038a24e3c1
I20260925 19:00:35.436524 7821 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:00:35.726506 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:35.726203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:52511)
0925 19:00:35.726419 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:35.726457 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
I20260925 19:00:35.726506 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:35.726213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:50563)
0925 19:00:35.726419 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:35.726457 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260925 19:00:35.726558 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:35.726289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:38869)
0925 19:00:35.726471 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:35.726521 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":55,"threads_started":1}
I20260925 19:00:36.727162 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:36.726930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:33153)
0925 19:00:36.727069 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:36.727109 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":52,"threads_started":1}
I20260925 19:00:36.727226 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:36.726916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:50621)
0925 19:00:36.727141 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:36.727176 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":55,"threads_started":1}
I20260925 19:00:36.727312 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:36.727146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:48053)
0925 19:00:36.727243 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:36.727270 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":51,"threads_started":1}
I20260925 19:00:37.727777 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:37.727543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:40379)
0925 19:00:37.727683 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:37.727721 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260925 19:00:37.727829 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:37.727608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:59987)
0925 19:00:37.727746 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:37.727781 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":49,"threads_started":1}
I20260925 19:00:37.728026 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:37.727795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:49567)
0925 19:00:37.727935 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:37.727985 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":38,"threads_started":1}
I20260925 19:00:38.728411 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:38.728200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:50905)
0925 19:00:38.728319 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:38.728356 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260925 19:00:38.728461 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:38.728262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:43807)
0925 19:00:38.728380 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:38.728412 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260925 19:00:38.728701 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:38.728360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:51915)
0925 19:00:38.728519 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:38.728668 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":56,"threads_started":1}
I20260925 19:00:39.729072 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:39.728887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:45167)
0925 19:00:39.728986 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:39.729021 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":49,"threads_started":1}
I20260925 19:00:39.729236 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:39.729070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:46307)
0925 19:00:39.729171 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:39.729198 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":39,"threads_started":1}
I20260925 19:00:39.729264 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:39.728853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:42365)
0925 19:00:39.728987 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:39.729030 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260925 19:00:40.729707 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:40.729556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:38409)
0925 19:00:40.729652 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:40.729676 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":41,"threads_started":1}
I20260925 19:00:40.729703 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:40.729486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:36471)
0925 19:00:40.729601 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:40.729652 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260925 19:00:40.729786 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:40.729585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:46939)
0925 19:00:40.729691 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:40.729743 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260925 19:00:41.730463 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:41.730134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:46975)
0925 19:00:41.730360 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:41.730401 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":78,"threads_started":1}
I20260925 19:00:41.730530 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:41.730365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:40957)
0925 19:00:41.730463 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:41.730490 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":45,"threads_started":1}
I20260925 19:00:41.730595 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:41.730236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:39715)
0925 19:00:41.730464 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:41.730502 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":150,"threads_started":1}
I20260925 19:00:42.731164 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:42.730920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:53779)
0925 19:00:42.731068 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:42.731105 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260925 19:00:42.731240 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:42.730916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:51163)
0925 19:00:42.731133 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:42.731184 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260925 19:00:42.731359 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:42.731087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:41363)
0925 19:00:42.731281 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:42.731321 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":61,"threads_started":1}
I20260925 19:00:43.731846 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:43.731638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:38971)
0925 19:00:43.731756 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:43.731790 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260925 19:00:43.731874 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:43.731579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:57887)
0925 19:00:43.731736 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:43.731778 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":55,"threads_started":1}
I20260925 19:00:43.731956 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:43.731794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:50125)
0925 19:00:43.731884 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:43.731912 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":42,"threads_started":1}
I20260925 19:00:44.732515 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:44.732278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:59927)
0925 19:00:44.732417 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:44.732459 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260925 19:00:44.732528 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:44.732278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:44573)
0925 19:00:44.732456 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:44.732484 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260925 19:00:44.732528 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:44.732278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:49335)
0925 19:00:44.732449 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:44.732484 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260925 19:00:45.486788 7854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5986923f01141dfa7d222038a24e3c1)
I20260925 19:00:45.486876 7854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5986923f01141dfa7d222038a24e3c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41805 } } peers { permanent_uuid: "3580c5192ad74c98bbeb4254f62d1fac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39997 } attrs { promote: false } } peers { permanent_uuid: "fc5658ea063c468eb33f350dfa9ded66" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40113 } attrs { promote: true } }
I20260925 19:00:45.486996 7854 leader_election.cc:288] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805)
I20260925 19:00:45.487334 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:45.487157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:34910)
0925 19:00:45.487258 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:45.487298 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
W20260925 19:00:45.487416 7067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260925 19:00:45.487795 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:45.487691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.0.1:34916)
0925 19:00:45.487737 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:45.487761 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:00:45.487879 7067 leader_election.cc:334] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5986923f01141dfa7d222038a24e3c1 (127.0.98.62:41805): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
I20260925 19:00:45.487939 7067 leader_election.cc:302] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3580c5192ad74c98bbeb4254f62d1fac; no voters: b5986923f01141dfa7d222038a24e3c1
I20260925 19:00:45.488023 7854 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3580c5192ad74c98bbeb4254f62d1fac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:00:45.733134 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:45.732936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:43761)
0925 19:00:45.733042 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:45.733078 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":42,"threads_started":1}
I20260925 19:00:45.733135 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:45.732905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:40757)
0925 19:00:45.733044 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:45.733083 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260925 19:00:45.733198 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:45.732906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:58645)
0925 19:00:45.733043 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:45.733078 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260925 19:00:46.733770 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:46.733549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:38307)
0925 19:00:46.733667 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:46.733706 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260925 19:00:46.733795 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:46.733593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:44485)
0925 19:00:46.733710 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:46.733745 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260925 19:00:46.734297 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:46.733744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:59407)
0925 19:00:46.734068 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:46.734103 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":52,"threads_started":1}
I20260925 19:00:47.734546 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:47.734286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:39207)
0925 19:00:47.734449 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:47.734495 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":57,"threads_started":1}
I20260925 19:00:47.734546 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:47.734310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:55553)
0925 19:00:47.734479 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:47.734505 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":53,"threads_started":1}
I20260925 19:00:47.734809 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:47.734619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:55277)
0925 19:00:47.734705 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:47.734741 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":44,"threads_started":1}
I20260925 19:00:48.735270 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:48.734900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:44601)
0925 19:00:48.735172 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:48.735213 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":92,"threads_started":1}
I20260925 19:00:48.735270 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:48.734998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:49769)
0925 19:00:48.735117 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:48.735221 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":44,"threads_started":1}
I20260925 19:00:48.735332 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:48.735130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:53017)
0925 19:00:48.735258 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:48.735282 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260925 19:00:49.735911 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:49.735668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:55183)
0925 19:00:49.735798 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:49.735850 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260925 19:00:49.736001 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:49.735668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:53243)
0925 19:00:49.735927 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:49.735960 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":60,"threads_started":1}
I20260925 19:00:49.735994 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:49.735680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:41683)
0925 19:00:49.735909 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:49.735949 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":47,"threads_started":1}
I20260925 19:00:50.736608 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:50.736390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:35929)
0925 19:00:50.736517 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:50.736556 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260925 19:00:50.736608 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:50.736435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:60077)
0925 19:00:50.736542 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:50.736569 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260925 19:00:50.736788 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:50.736435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:48377)
0925 19:00:50.736720 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:50.736749 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":55,"threads_started":1}
I20260925 19:00:51.737416 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:51.737249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:37591)
0925 19:00:51.737360 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:51.737383 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260925 19:00:51.737416 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:51.737041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:45371)
0925 19:00:51.737315 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:51.737354 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":51,"threads_started":1}
I20260925 19:00:51.737433 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:51.737030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:41019)
0925 19:00:51.737345 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:51.737386 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":59,"threads_started":1}
I20260925 19:00:52.738036 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:52.737811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:54547)
0925 19:00:52.737942 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:52.737981 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260925 19:00:52.738183 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:52.737807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:60911)
0925 19:00:52.738112 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:52.738144 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":58,"threads_started":1}
I20260925 19:00:52.738108 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:52.737819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:54245)
0925 19:00:52.737951 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:52.737987 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260925 19:00:53.738763 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:53.738555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:46821)
0925 19:00:53.738664 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:53.738705 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260925 19:00:53.738843 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:53.738530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:42075)
0925 19:00:53.738759 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:53.738797 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":62,"threads_started":1}
I20260925 19:00:53.738917 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:53.738590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:36135)
0925 19:00:53.738830 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:53.738877 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":40,"threads_started":1}
I20260925 19:00:54.739611 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:54.739314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.1:55901)
0925 19:00:54.739441 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:54.739479 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260925 19:00:54.739622 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:54.739314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.3:47261)
0925 19:00:54.739555 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:54.739587 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":60,"threads_started":1}
I20260925 19:00:54.739648 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:54.739313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41805 (local address 127.0.98.2:45443)
0925 19:00:54.739516 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:54.739554 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":65,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:41805,127.0.98.61:39997,127.0.98.60:40113, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:41805,127.0.98.61:39997,127.0.98.60:40113, num_attempts: 1)
I20260925 19:00:54.879021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 7233
I20260925 19:00:54.883994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 7456
I20260925 19:00:54.889052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 7592
I20260925 19:00:54.893370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 6948
I20260925 19:00:54.897480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 7135
2026-09-25T19:00:54Z chronyd exiting
I20260925 19:00:54.913059 392 test_util.cc:175] -----------------------------------------------
I20260925 19:00:54.913105 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790362794046639-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32901 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:54 dist-test-slave-49bz krb5kdc[7889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:54 dist-test-slave-49bz krb5kdc[7889](info): set up 2 sockets
Sep 25 19:00:54 dist-test-slave-49bz krb5kdc[7889](info): commencing operation
krb5kdc: starting...
W20260925 19:00:55.898545 392 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:55 dist-test-slave-49bz krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 19:00:56.014381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40119
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:56.094769 7902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:56.094942 7902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:56.094961 7902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:56.095999 7902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:56.096040 7902 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 19:00:56.096052 7902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:56.096063 7902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:56.096074 7902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:56.096086 7902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:56.097396 7902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40119
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:56.097594 7902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:56.097779 7902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:56.102495 7908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:56.102535 7907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:56.102543 7902 server_base.cc:1034] running on GCE node
W20260925 19:00:56.102522 7910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:56.102936 7902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:56.103161 7902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:56.104295 7902 hybrid_clock.cc:648] HybridClock initialized: now 1790362856104275 us; error 35 us; skew 500 ppm
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:56.106549 7902 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 19:00:56.106988 7902 webserver.cc:460] Webserver started at http://127.0.98.62:32967/ using document root <none> and password file <none>
I20260925 19:00:56.107215 7902 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:56.107257 7902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:56.107395 7902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:56.108418 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "f60f73e2cfd84ed387837d19b23776f2"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "6ffef846ae39920b2fef26e56ebe3fc9"
server_key_iv: "f7035b2e29e01a4d5444becbec961e1f"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.108752 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "f60f73e2cfd84ed387837d19b23776f2"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "6ffef846ae39920b2fef26e56ebe3fc9"
server_key_iv: "f7035b2e29e01a4d5444becbec961e1f"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.109998 7902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:00:56.110736 7917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.110900 7902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.111016 7902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "f60f73e2cfd84ed387837d19b23776f2"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "6ffef846ae39920b2fef26e56ebe3fc9"
server_key_iv: "f7035b2e29e01a4d5444becbec961e1f"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.111083 7902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:56.121656 7902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:56.123327 7902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:56.123478 7902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:56.127817 7902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40119
I20260925 19:00:56.127857 7979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40119 every 8 connection(s)
I20260925 19:00:56.128177 7902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:56.128692 7980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:56.130780 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2: Bootstrap starting.
I20260925 19:00:56.131299 7980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:56.131578 7980 log.cc:824] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:56.132189 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2: No bootstrap required, opened a new log
I20260925 19:00:56.134420 7980 raft_consensus.cc:348] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60f73e2cfd84ed387837d19b23776f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260925 19:00:56.134542 7980 raft_consensus.cc:374] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:56.134563 7980 raft_consensus.cc:729] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f60f73e2cfd84ed387837d19b23776f2, State: Initialized, Role: FOLLOWER
I20260925 19:00:56.134706 7980 consensus_queue.cc:260] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60f73e2cfd84ed387837d19b23776f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260925 19:00:56.134761 7980 raft_consensus.cc:388] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:56.134796 7980 raft_consensus.cc:482] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:56.134840 7980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:56.135406 7980 raft_consensus.cc:504] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60f73e2cfd84ed387837d19b23776f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260925 19:00:56.135507 7980 leader_election.cc:302] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f60f73e2cfd84ed387837d19b23776f2; no voters:
I20260925 19:00:56.135684 7980 leader_election.cc:288] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:56.135720 7983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:56.135892 7983 raft_consensus.cc:686] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [term 1 LEADER]: Becoming Leader. State: Replica: f60f73e2cfd84ed387837d19b23776f2, State: Running, Role: LEADER
I20260925 19:00:56.135931 7980 sys_catalog.cc:564] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:56.136032 7983 consensus_queue.cc:237] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60f73e2cfd84ed387837d19b23776f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260925 19:00:56.136415 7984 sys_catalog.cc:455] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f60f73e2cfd84ed387837d19b23776f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60f73e2cfd84ed387837d19b23776f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } } }
I20260925 19:00:56.136425 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f60f73e2cfd84ed387837d19b23776f2. Latest consensus state: current_term: 1 leader_uuid: "f60f73e2cfd84ed387837d19b23776f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f60f73e2cfd84ed387837d19b23776f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } } }
I20260925 19:00:56.136606 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:56.136592 7984 sys_catalog.cc:458] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:56.136794 7992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:56.137298 7992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:56.138376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 7902
I20260925 19:00:56.138486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:56.138707 392 external_mini_cluster.cc:1428] Setting key 45d4d26c8413b82105c50ccf449415e3
I20260925 19:00:56.138903 7992 catalog_manager.cc:1347] Generated new cluster ID: 387c90da3c1f40c79ea4dfda77686fd0
I20260925 19:00:56.138958 7992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:56.149556 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.139287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.0.98.62:40119)
0925 19:00:56.139413 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.139421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.139452 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:00:56.139550 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.139556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.139603 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.139716 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:56.140451 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.141770 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.142882 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.143494 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.143591 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.146197 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:56.146200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:56.146201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:56.146211 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:56.148075 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.148434 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.148437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.148438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.148542 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.148730 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.148731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.148731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.148996 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:56.149139 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.149201 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.149319 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260925 19:00:56.159466 7992 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:56.159932 7992 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:56.167621 7992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2: Generated new TSK 0
I20260925 19:00:56.167790 7992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 19:00:56.197220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40119
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:56.275996 8006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:56.276182 8006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:56.276206 8006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:56.277274 8006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:56.277308 8006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:56.277328 8006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:00:56.277344 8006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:56.278717 8006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:56.278910 8006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:56.279078 8006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:56.283798 8012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:56.283798 8014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:56.283860 8011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:56.283965 8006 server_base.cc:1034] running on GCE node
I20260925 19:00:56.284288 8006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:56.284512 8006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:56.285679 8006 hybrid_clock.cc:648] HybridClock initialized: now 1790362856285663 us; error 28 us; skew 500 ppm
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:56.287998 8006 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 19:00:56.288421 8006 webserver.cc:460] Webserver started at http://127.0.98.1:37047/ using document root <none> and password file <none>
I20260925 19:00:56.288638 8006 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:56.288707 8006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:56.288854 8006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:56.289866 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "18048183b997408c812b70c748f2c8a9"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "64f717873dc328afe1118301e496d9c3"
server_key_iv: "92ccca01470cc96f633712b32cf9cd5a"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.290225 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "18048183b997408c812b70c748f2c8a9"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "64f717873dc328afe1118301e496d9c3"
server_key_iv: "92ccca01470cc96f633712b32cf9cd5a"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.291543 8006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:56.292287 8021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.292462 8006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:56.292567 8006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "18048183b997408c812b70c748f2c8a9"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "64f717873dc328afe1118301e496d9c3"
server_key_iv: "92ccca01470cc96f633712b32cf9cd5a"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.292644 8006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:56.310786 8006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:56.312574 8006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:56.312732 8006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:56.313334 8006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:56.313406 8006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.313448 8006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:56.313476 8006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.320268 8006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34875
I20260925 19:00:56.320336 8136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34875 every 8 connection(s)
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:56.320698 8006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:00:56.321376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 8006
I20260925 19:00:56.321492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:56.321677 392 external_mini_cluster.cc:1428] Setting key 4edd3dad17e90285cb3ba92bcebcf3e9
I20260925 19:00:56.324079 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.313717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41147 (local address 127.0.98.62:40119)
0925 19:00:56.313771 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.313773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.314023 (+ 250us) server_negotiation.cc:407] Connection header received
0925 19:00:56.314558 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.314559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.314572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.314655 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:56.315482 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.316315 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.317911 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.318314 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.318399 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.321032 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:56.321034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:56.321034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:56.321044 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:56.322682 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.323130 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.323131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.323132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.323188 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.323445 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.323446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.323446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.323562 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:56.323638 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.323904 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.323988 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:00:56.324249 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.313728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.1:41147)
0925 19:00:56.313872 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.313919 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.314460 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.314703 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.314706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.315092 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:56.315378 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.315385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.316436 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.316437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.317809 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.317812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.318413 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.318414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.318527 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.319078 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:56.319088 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:56.320911 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:56.322781 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.322788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.322790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.323035 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.323261 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.323262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.323262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.323372 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.323628 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:56.323634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:56.323768 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.323868 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.324025 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260925 19:00:56.324764 8137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40119
I20260925 19:00:56.324855 8137 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:56.325042 8137 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260925 19:00:56.325645 7933 ts_manager.cc:194] Registered new tserver with Master: 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875)
I20260925 19:00:56.326297 7933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41147
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 19:00:56.332228 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.326537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43865 (local address 127.0.98.62:40119)
0925 19:00:56.326576 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.326578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.326840 (+ 262us) server_negotiation.cc:407] Connection header received
0925 19:00:56.326888 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.326889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.326899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.326965 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:56.327376 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.327813 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.328848 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.329027 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.329078 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.330015 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:56.330016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:56.330016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:56.330022 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:56.331417 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.331703 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.331704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.331704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.331751 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.331892 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.331892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.331893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.331977 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:56.332019 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.332121 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.332175 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:56.332268 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.326494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.1:43865)
0925 19:00:56.326752 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.326769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.326832 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.326991 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.326992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.327221 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:56.327297 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.327300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.327908 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.327909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.328764 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.328768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.329104 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.329105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.329172 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.329559 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:56.329564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:56.329933 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:56.331488 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.331489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.331490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.331642 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.331807 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.331808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.331808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.331855 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.332023 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:56.332023 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:56.332060 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.332108 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.332177 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 19:00:56.347024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40119
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:56.429654 8146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:56.429841 8146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:56.429864 8146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:56.431042 8146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:56.431080 8146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:56.431102 8146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:00:56.431118 8146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:00:56.432472 8146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:56.432653 8146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:56.432806 8146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:56.437407 8154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:56.437481 8152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:56.437417 8151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:56.437638 8146 server_base.cc:1034] running on GCE node
I20260925 19:00:56.437849 8146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:56.438088 8146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:56.439224 8146 hybrid_clock.cc:648] HybridClock initialized: now 1790362856439210 us; error 30 us; skew 500 ppm
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:56.441555 8146 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 19:00:56.441988 8146 webserver.cc:460] Webserver started at http://127.0.98.2:35057/ using document root <none> and password file <none>
I20260925 19:00:56.442201 8146 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:56.442304 8146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:56.442451 8146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:56.443544 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "0310a7fbe57d49c7bd592dbeef606a21"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "7125646ce68e9e793ba2a6392805e8d9"
server_key_iv: "c34f4f55d3dc2ce8a7f7a7ac32955b2b"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.443881 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0310a7fbe57d49c7bd592dbeef606a21"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "7125646ce68e9e793ba2a6392805e8d9"
server_key_iv: "c34f4f55d3dc2ce8a7f7a7ac32955b2b"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.445087 8146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:56.445767 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.445935 8146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:56.446025 8146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "0310a7fbe57d49c7bd592dbeef606a21"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "7125646ce68e9e793ba2a6392805e8d9"
server_key_iv: "c34f4f55d3dc2ce8a7f7a7ac32955b2b"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.446102 8146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:56.462986 8146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:56.464675 8146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:56.464823 8146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:56.465363 8146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:56.465415 8146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.465442 8146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:56.465497 8146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:56.472157 8146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35951
I20260925 19:00:56.472227 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35951 every 8 connection(s)
I20260925 19:00:56.472524 8146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:00:56.474645 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.465699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46561 (local address 127.0.98.62:40119)
0925 19:00:56.465772 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.465775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.465961 (+ 186us) server_negotiation.cc:407] Connection header received
0925 19:00:56.466465 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.466466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.466479 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.466538 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:56.467110 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.467879 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.469539 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.469889 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.469976 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.472010 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:56.472012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:56.472012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:56.472020 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:56.473493 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.473826 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.473827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.473827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.473891 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.474099 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.474100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.474100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.474194 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:56.474241 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.474529 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.474586 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 19:00:56.474767 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.465700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.2:46561)
0925 19:00:56.465856 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.465891 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.466344 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.466599 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.466603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.466837 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:56.467015 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.467019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.468007 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.468008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.469432 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.469435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.469992 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.469994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.470105 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.470577 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:56.470585 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:56.471918 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:56.473562 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.473566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.473567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.473762 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.473958 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.473959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.473959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.474046 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.474297 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:56.474302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:56.474420 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.474499 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.474599 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260925 19:00:56.475237 8277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40119
I20260925 19:00:56.475315 8277 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:56.475524 8277 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260925 19:00:56.475926 7934 ts_manager.cc:194] Registered new tserver with Master: 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951)
I20260925 19:00:56.476378 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46561
I20260925 19:00:56.481187 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.476212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.2:45157)
0925 19:00:56.476349 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.476358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.476411 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.476534 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.476535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.476727 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:56.476799 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.476800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.477467 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.477467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.477938 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.477939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.478214 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.478215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.478300 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.478621 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:56.478624 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:56.478856 (+ 232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:56.480346 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.480347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.480347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.480547 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.480720 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.480721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.480721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.480770 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.480970 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:56.480970 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:56.481008 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.481059 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.481124 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260925 19:00:56.481225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 8146
I20260925 19:00:56.481264 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.476263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45157 (local address 127.0.98.62:40119)
0925 19:00:56.476302 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.476304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.476421 (+ 117us) server_negotiation.cc:407] Connection header received
0925 19:00:56.476446 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.476447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.476456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.476519 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:56.476873 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.477383 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.478001 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.478158 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.478202 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.478928 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:56.478929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:56.478929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:56.478935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:56.480253 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.480619 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.480619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.480620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.480666 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.480807 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.480807 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.480807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.480920 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:56.480965 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.481096 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.481169 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:56.481331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:00:56.481499 392 external_mini_cluster.cc:1428] Setting key 5b0f4e46cca4b45311888c13022fc2f3
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 19:00:56.504887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40119
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:56.581400 8286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:56.581598 8286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:56.581633 8286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:56.582669 8286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:56.582727 8286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:56.582758 8286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 19:00:56.582793 8286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:00:56.584220 8286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:56.584481 8286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:56.584695 8286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:56.589332 8292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:56.589321 8294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:56.589350 8291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:56.589425 8286 server_base.cc:1034] running on GCE node
I20260925 19:00:56.589726 8286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:56.589982 8286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:56.591144 8286 hybrid_clock.cc:648] HybridClock initialized: now 1790362856591123 us; error 33 us; skew 500 ppm
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:56.593335 8286 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 19:00:56.593729 8286 webserver.cc:460] Webserver started at http://127.0.98.3:41255/ using document root <none> and password file <none>
I20260925 19:00:56.593946 8286 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:56.594013 8286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:56.594141 8286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:56.595103 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "39b6a652fa314ab6a553fd3b14d153fb"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "8ec469761ff576adf5bb927037eb1c17"
server_key_iv: "d6fa1dbaf6ef78daeff4e1b0c8b7b916"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.595471 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "39b6a652fa314ab6a553fd3b14d153fb"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "8ec469761ff576adf5bb927037eb1c17"
server_key_iv: "d6fa1dbaf6ef78daeff4e1b0c8b7b916"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.596746 8286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:00:56.597363 8301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.597530 8286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:56.597627 8286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "39b6a652fa314ab6a553fd3b14d153fb"
format_stamp: "Formatted at 2026-09-25 19:00:56 on dist-test-slave-49bz"
server_key: "8ec469761ff576adf5bb927037eb1c17"
server_key_iv: "d6fa1dbaf6ef78daeff4e1b0c8b7b916"
server_key_version: "encryptionkey@0"
I20260925 19:00:56.597699 8286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:56.606405 8286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:56.607955 8286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:56.608103 8286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:56.608526 8286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:56.608579 8286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.608614 8286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:56.608639 8286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:56.615645 8286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40857
I20260925 19:00:56.615739 8416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40857 every 8 connection(s)
I20260925 19:00:56.615986 8286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:00:56.617813 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.609028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41835 (local address 127.0.98.62:40119)
0925 19:00:56.609073 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.609075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.609283 (+ 208us) server_negotiation.cc:407] Connection header received
0925 19:00:56.609757 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.609758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.609771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.609837 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:56.610899 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.611689 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.612661 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.612867 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.612947 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.615302 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:56.615303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:56.615304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:56.615312 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:56.616823 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.617154 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.617155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.617155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.617201 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.617349 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.617349 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.617350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.617436 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:56.617478 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.617701 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.617756 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:00:56.617928 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.609024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.3:41835)
0925 19:00:56.609177 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.609206 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.609633 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.609917 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.609921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.610505 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:56.610773 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.610780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.611775 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.611776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.612553 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.612555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.612964 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.612965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.613105 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.613634 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:56.613643 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:56.615206 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:56.616889 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.616892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.616894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.617090 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.617242 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.617243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.617243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.617316 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.617486 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:56.617490 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:56.617597 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.617671 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.617772 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260925 19:00:56.618384 8417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40119
I20260925 19:00:56.618461 8417 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:56.618629 8417 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260925 19:00:56.618850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 8286
I20260925 19:00:56.618959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:00:56.618991 7933 ts_manager.cc:194] Registered new tserver with Master: 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857)
I20260925 19:00:56.619107 392 external_mini_cluster.cc:1428] Setting key a4ee435c35df5c87df91b85a1dc1363d
I20260925 19:00:56.619439 7933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41835
I20260925 19:00:56.619989 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:00:56.624743 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.619430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37183 (local address 127.0.98.62:40119)
0925 19:00:56.619635 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.619638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.619646 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:56.619663 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.619663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.619671 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.619719 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:56.620152 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.620619 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.621266 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.621468 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.621547 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.622352 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:56.622354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:56.622354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:56.622360 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:56.623637 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.623974 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.623975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.623976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.624025 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.624263 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.624264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.624264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.624374 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:56.624449 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.624596 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.624660 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260925 19:00:56.624758 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.619287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.3:37183)
0925 19:00:56.619487 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.619504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.619545 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.619771 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.619772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.619999 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:56.620083 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.620085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.620706 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.620707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.621189 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.621190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.621550 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.621551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.621609 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.621995 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:56.621999 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:56.622234 (+ 235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:56.623733 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.623734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.623735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.623898 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.624127 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.624127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.624128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.624183 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.624453 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:56.624454 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:56.624493 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.624574 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.624661 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":62,"threads_started":1}
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362856, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 19:00:56.640594 8003 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:00:56.635316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.0.98.62:40119)
0925 19:00:56.635368 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.635370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.635380 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:00:56.635471 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.635472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.635479 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.635534 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:56.636136 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.636671 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.637473 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.637659 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.637739 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.640524 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38370: BlockingRecv error: recv got EOF from 127.0.0.1:38370 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362856, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:56.647395 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.641350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.0.98.62:40119)
0925 19:00:56.641392 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.641393 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.641498 (+ 105us) server_negotiation.cc:407] Connection header received
0925 19:00:56.641532 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.641533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.641540 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.641584 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:56.641973 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.642451 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.643038 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.643217 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.643261 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.645216 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:56.645217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:56.645218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:56.645223 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:56.646613 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.646881 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.646882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.646882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.646924 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.647066 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.647066 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.647066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.647142 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:56.647202 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.647281 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.647346 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:00:56.649060 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:00:56.650833 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:56.656490 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.652836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40857 (local address 127.0.0.1:42654)
0925 19:00:56.652931 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.652969 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.653030 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.653575 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.653576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.653577 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:56.653627 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.653629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.654548 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.654549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.655336 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.655337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.656228 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.656229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.656300 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.656312 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.656380 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.656419 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":52,"threads_started":1}
I20260925 19:00:56.656538 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.653177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.0.98.3:40857)
0925 19:00:56.653336 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.653340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.653349 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:56.653388 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.653389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.653400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.653509 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:56.653704 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.654471 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.655450 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.656146 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.656217 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.656284 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.656405 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.656440 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260925 19:00:56.656817 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.652843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35951 (local address 127.0.0.1:52874)
0925 19:00:56.652983 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.652993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.653044 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.653338 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.653341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.653344 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:56.653413 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.653415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.654413 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.654414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.655718 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.655720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.656607 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.656608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.656674 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.656678 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.656724 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.656766 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":53,"threads_started":1}
I20260925 19:00:56.657111 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.652940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.0.98.2:35951)
0925 19:00:56.653152 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.653156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.653165 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:56.653206 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.653208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.653219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.653312 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:56.653515 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.654342 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.655790 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.656449 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.656511 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.656572 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.656969 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.657003 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260925 19:00:56.657151 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 42e38924b7d94301956c675a4084ec52 (DEFAULT_TABLE table=test-table [id=401389ba90d64a5d8cd5c5f30c1dd23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:56.657580 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e38924b7d94301956c675a4084ec52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:56.657732 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 42e38924b7d94301956c675a4084ec52 (DEFAULT_TABLE table=test-table [id=401389ba90d64a5d8cd5c5f30c1dd23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:56.658062 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e38924b7d94301956c675a4084ec52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:56.658408 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.653047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.0.1:37626)
0925 19:00:56.653581 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.653591 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.653635 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.654774 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.654774 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.654777 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:56.654807 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.654809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.655863 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.655865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.656945 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.656947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.658152 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.658153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.658215 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.658218 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.658303 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.658340 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":75,"threads_started":1}
I20260925 19:00:56.658638 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.653419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.0.98.1:34875)
0925 19:00:56.654617 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.654621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.654630 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:56.654665 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.654667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.654680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.654763 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:56.654869 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.655787 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.657044 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.657977 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.658047 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.658118 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.658324 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.658365 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1164,"thread_start_us":83,"threads_started":1}
I20260925 19:00:56.660585 8442 tablet_bootstrap.cc:492] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: Bootstrap starting.
I20260925 19:00:56.661209 8443 tablet_bootstrap.cc:492] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: Bootstrap starting.
I20260925 19:00:56.661453 8442 tablet_bootstrap.cc:654] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:56.661345 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 42e38924b7d94301956c675a4084ec52 (DEFAULT_TABLE table=test-table [id=401389ba90d64a5d8cd5c5f30c1dd23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:56.661671 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e38924b7d94301956c675a4084ec52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:56.661736 8442 log.cc:824] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:56.661777 8443 tablet_bootstrap.cc:654] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:56.662118 8443 log.cc:824] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:56.662544 8442 tablet_bootstrap.cc:492] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: No bootstrap required, opened a new log
I20260925 19:00:56.662611 8442 ts_tablet_manager.cc:1397] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:56.662799 8443 tablet_bootstrap.cc:492] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: No bootstrap required, opened a new log
I20260925 19:00:56.662874 8443 ts_tablet_manager.cc:1397] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:00:56.664499 8443 raft_consensus.cc:348] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } }
I20260925 19:00:56.664635 8443 raft_consensus.cc:374] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:56.664673 8443 raft_consensus.cc:729] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0310a7fbe57d49c7bd592dbeef606a21, State: Initialized, Role: FOLLOWER
I20260925 19:00:56.664675 8442 raft_consensus.cc:348] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } }
I20260925 19:00:56.664846 8442 raft_consensus.cc:374] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:56.664805 8443 consensus_queue.cc:260] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } }
I20260925 19:00:56.664896 8442 raft_consensus.cc:729] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b6a652fa314ab6a553fd3b14d153fb, State: Initialized, Role: FOLLOWER
I20260925 19:00:56.665030 8442 consensus_queue.cc:260] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } }
I20260925 19:00:56.665091 8277 heartbeater.cc:503] Master 127.0.98.62:40119 was elected leader, sending a full tablet report...
I20260925 19:00:56.665277 8417 heartbeater.cc:503] Master 127.0.98.62:40119 was elected leader, sending a full tablet report...
I20260925 19:00:56.665335 8443 ts_tablet_manager.cc:1428] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:00:56.665512 8442 ts_tablet_manager.cc:1428] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:00:56.665695 8447 tablet_bootstrap.cc:492] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: Bootstrap starting.
I20260925 19:00:56.666420 8447 tablet_bootstrap.cc:654] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:56.666705 8447 log.cc:824] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:56.667305 8447 tablet_bootstrap.cc:492] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: No bootstrap required, opened a new log
I20260925 19:00:56.667353 8447 ts_tablet_manager.cc:1397] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:56.668764 8447 raft_consensus.cc:348] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } }
I20260925 19:00:56.668871 8447 raft_consensus.cc:374] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:56.668890 8447 raft_consensus.cc:729] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18048183b997408c812b70c748f2c8a9, State: Initialized, Role: FOLLOWER
I20260925 19:00:56.669023 8447 consensus_queue.cc:260] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } }
I20260925 19:00:56.669238 8447 ts_tablet_manager.cc:1428] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:00:56.669275 8137 heartbeater.cc:503] Master 127.0.98.62:40119 was elected leader, sending a full tablet report...
I20260925 19:00:56.715065 8446 raft_consensus.cc:482] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:56.715181 8446 raft_consensus.cc:504] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } }
I20260925 19:00:56.715507 8446 leader_election.cc:288] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857), 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875)
I20260925 19:00:56.718456 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.715781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.98.2:52443)
0925 19:00:56.715931 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.715940 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.715982 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.716142 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.716143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.716144 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:56.716223 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.716225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.716877 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.716877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.717584 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.717586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.718114 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.718115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.718196 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.718204 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.718292 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.718354 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260925 19:00:56.718508 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.715734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58765 (local address 127.0.98.3:40857)
0925 19:00:56.715796 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.715798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.715807 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:56.715837 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.715837 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.715848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.715898 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:56.716058 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.716581 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.717582 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.718074 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.718140 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.718149 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.718390 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.718437 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:00:56.718518 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.715957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52443 (local address 127.0.98.1:34875)
0925 19:00:56.716000 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.716002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.716035 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:00:56.716056 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.716057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.716066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.716109 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:56.716287 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.716802 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.717674 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.718027 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.718114 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.718125 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.718295 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.718351 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:00:56.718724 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.715672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40857 (local address 127.0.98.2:58765)
0925 19:00:56.715732 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.715743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.715798 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.715916 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.715917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.715919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:56.715981 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.715983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.716666 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.716667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.717500 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.717502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.718165 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.718166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.718296 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.718300 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.718365 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.718402 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":21}
I20260925 19:00:56.718904 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e38924b7d94301956c675a4084ec52" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18048183b997408c812b70c748f2c8a9" is_pre_election: true
I20260925 19:00:56.719046 8091 raft_consensus.cc:2454] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0310a7fbe57d49c7bd592dbeef606a21 in term 0.
I20260925 19:00:56.719017 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e38924b7d94301956c675a4084ec52" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b6a652fa314ab6a553fd3b14d153fb" is_pre_election: true
I20260925 19:00:56.719125 8371 raft_consensus.cc:2454] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0310a7fbe57d49c7bd592dbeef606a21 in term 0.
I20260925 19:00:56.719264 8162 leader_election.cc:302] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0310a7fbe57d49c7bd592dbeef606a21, 18048183b997408c812b70c748f2c8a9; no voters:
I20260925 19:00:56.719391 8446 raft_consensus.cc:2781] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:56.719455 8446 raft_consensus.cc:482] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:56.719478 8446 raft_consensus.cc:3037] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:56.720315 8446 raft_consensus.cc:504] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } }
I20260925 19:00:56.720433 8446 leader_election.cc:288] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 election: Requested vote from peers 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857), 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875)
I20260925 19:00:56.720563 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e38924b7d94301956c675a4084ec52" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b6a652fa314ab6a553fd3b14d153fb"
I20260925 19:00:56.720598 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e38924b7d94301956c675a4084ec52" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18048183b997408c812b70c748f2c8a9"
I20260925 19:00:56.720639 8371 raft_consensus.cc:3037] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:56.720659 8091 raft_consensus.cc:3037] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:56.721424 8371 raft_consensus.cc:2454] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0310a7fbe57d49c7bd592dbeef606a21 in term 1.
I20260925 19:00:56.721433 8091 raft_consensus.cc:2454] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0310a7fbe57d49c7bd592dbeef606a21 in term 1.
I20260925 19:00:56.721606 8162 leader_election.cc:302] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0310a7fbe57d49c7bd592dbeef606a21, 39b6a652fa314ab6a553fd3b14d153fb; no voters:
I20260925 19:00:56.721698 8446 raft_consensus.cc:2781] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:56.721818 8446 raft_consensus.cc:686] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 LEADER]: Becoming Leader. State: Replica: 0310a7fbe57d49c7bd592dbeef606a21, State: Running, Role: LEADER
I20260925 19:00:56.721910 8446 consensus_queue.cc:237] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } }
I20260925 19:00:56.722659 7933 catalog_manager.cc:5495] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0310a7fbe57d49c7bd592dbeef606a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } health_report { overall_health: HEALTHY } } }
W20260925 19:00:56.723258 8278 tablet.cc:2369] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:00:56.728009 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.725982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.0.98.2:35951)
0925 19:00:56.726044 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.726045 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.726072 (+ 27us) server_negotiation.cc:407] Connection header received
0925 19:00:56.726103 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.726104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.726118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.726164 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:56.726303 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.726719 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.727402 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.727571 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.727615 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.727742 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:56.727827 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:56.727876 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.727906 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.727925 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:00:56.729393 8371 raft_consensus.cc:1261] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Refusing update from remote peer 0310a7fbe57d49c7bd592dbeef606a21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:56.729393 8091 raft_consensus.cc:1261] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Refusing update from remote peer 0310a7fbe57d49c7bd592dbeef606a21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:56.729660 8452 consensus_queue.cc:1035] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:56.729743 8446 consensus_queue.cc:1035] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:56.732096 8455 mvcc.cc:205] Tried to move back new op lower bound from 7333326261161218048 to 7333326261133512704. Current Snapshot: MvccSnapshot[applied={T|T < 7333326261161218048}]
I20260925 19:00:56.732544 8456 mvcc.cc:205] Tried to move back new op lower bound from 7333326261161218048 to 7333326261133512704. Current Snapshot: MvccSnapshot[applied={T|T < 7333326261161218048}]
I20260925 19:00:56.734570 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:56.734658 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:56.735284 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2: Sending DeleteTablet for 3 replicas of tablet 42e38924b7d94301956c675a4084ec52
I20260925 19:00:56.735569 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e38924b7d94301956c675a4084ec52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42654
I20260925 19:00:56.735582 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e38924b7d94301956c675a4084ec52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37626
I20260925 19:00:56.735625 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e38924b7d94301956c675a4084ec52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52874
I20260925 19:00:56.735867 8466 tablet_replica.cc:330] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: stopping tablet replica
I20260925 19:00:56.735965 8466 raft_consensus.cc:2229] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:56.735954 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 19:00:56.736075 8466 raft_consensus.cc:2258] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 19:00:56.736088 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:56.736482 8466 ts_tablet_manager.cc:1905] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:56.736600 8465 tablet_replica.cc:330] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: stopping tablet replica
I20260925 19:00:56.736678 8465 raft_consensus.cc:2229] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:56.736764 8465 raft_consensus.cc:2258] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:56.737147 8465 ts_tablet_manager.cc:1905] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:56.737326 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 62f538bfa97347eb90be232893a8fae5 (DEFAULT_TABLE table=test-table [id=5c8447ee39aa4514b1006a3c56df9e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:56.737361 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 62f538bfa97347eb90be232893a8fae5 (DEFAULT_TABLE table=test-table [id=5c8447ee39aa4514b1006a3c56df9e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:56.737462 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f538bfa97347eb90be232893a8fae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:56.737458 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f538bfa97347eb90be232893a8fae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:56.738194 8466 ts_tablet_manager.cc:1918] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:56.738282 8466 log.cc:1196] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/42e38924b7d94301956c675a4084ec52
I20260925 19:00:56.738559 8466 ts_tablet_manager.cc:1939] T 42e38924b7d94301956c675a4084ec52 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting consensus metadata
I20260925 19:00:56.738751 8447 tablet_bootstrap.cc:492] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: Bootstrap starting.
I20260925 19:00:56.738762 8442 tablet_bootstrap.cc:492] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: Bootstrap starting.
I20260925 19:00:56.738869 8465 ts_tablet_manager.cc:1918] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:56.738929 8465 log.cc:1196] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/42e38924b7d94301956c675a4084ec52
I20260925 19:00:56.739122 8465 ts_tablet_manager.cc:1939] T 42e38924b7d94301956c675a4084ec52 P 18048183b997408c812b70c748f2c8a9: Deleting consensus metadata
I20260925 19:00:56.739216 8447 tablet_bootstrap.cc:654] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:56.739264 8464 tablet_replica.cc:330] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: stopping tablet replica
I20260925 19:00:56.739377 8464 raft_consensus.cc:2229] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:56.739475 8442 tablet_bootstrap.cc:654] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:56.739630 7918 catalog_manager.cc:4841] TS 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875): tablet 42e38924b7d94301956c675a4084ec52 (table test-table [id=401389ba90d64a5d8cd5c5f30c1dd23a]) successfully deleted
I20260925 19:00:56.739920 8447 tablet_bootstrap.cc:492] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: No bootstrap required, opened a new log
I20260925 19:00:56.739954 7921 catalog_manager.cc:4841] TS 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951): tablet 42e38924b7d94301956c675a4084ec52 (table test-table [id=401389ba90d64a5d8cd5c5f30c1dd23a]) successfully deleted
I20260925 19:00:56.739970 8447 ts_tablet_manager.cc:1397] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:56.739993 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 62f538bfa97347eb90be232893a8fae5 (DEFAULT_TABLE table=test-table [id=5c8447ee39aa4514b1006a3c56df9e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:56.740082 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f538bfa97347eb90be232893a8fae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:56.740123 8442 tablet_bootstrap.cc:492] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: No bootstrap required, opened a new log
I20260925 19:00:56.740108 8447 raft_consensus.cc:348] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.740187 8447 raft_consensus.cc:374] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:56.740188 8442 ts_tablet_manager.cc:1397] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:56.740209 8447 raft_consensus.cc:729] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18048183b997408c812b70c748f2c8a9, State: Initialized, Role: FOLLOWER
I20260925 19:00:56.740247 8447 consensus_queue.cc:260] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.740319 8447 ts_tablet_manager.cc:1428] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.740319 8442 raft_consensus.cc:348] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.740376 8442 raft_consensus.cc:374] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:56.740394 8442 raft_consensus.cc:729] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b6a652fa314ab6a553fd3b14d153fb, State: Initialized, Role: FOLLOWER
I20260925 19:00:56.740432 8442 consensus_queue.cc:260] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.740499 8442 ts_tablet_manager.cc:1428] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.741245 8443 tablet_bootstrap.cc:492] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21: Bootstrap starting.
I20260925 19:00:56.741675 8443 tablet_bootstrap.cc:654] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:56.742300 8443 tablet_bootstrap.cc:492] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21: No bootstrap required, opened a new log
I20260925 19:00:56.742347 8443 ts_tablet_manager.cc:1397] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 19:00:56.742467 8443 raft_consensus.cc:348] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.742530 8443 raft_consensus.cc:374] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:56.742549 8443 raft_consensus.cc:729] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0310a7fbe57d49c7bd592dbeef606a21, State: Initialized, Role: FOLLOWER
I20260925 19:00:56.742594 8443 consensus_queue.cc:260] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.742668 8443 ts_tablet_manager.cc:1428] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.743038 8450 raft_consensus.cc:482] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:56.743108 8450 raft_consensus.cc:504] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.743355 8450 leader_election.cc:288] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951), 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857)
I20260925 19:00:56.746160 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.743524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40393 (local address 127.0.98.2:35951)
0925 19:00:56.743559 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.743561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.743688 (+ 127us) server_negotiation.cc:407] Connection header received
0925 19:00:56.743788 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.743789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.743797 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.743841 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:56.744010 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.744462 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.745467 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.745818 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.745895 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.745905 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.746069 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.746100 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:00:56.746177 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.743499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35951 (local address 127.0.98.1:40393)
0925 19:00:56.743628 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.743638 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.743713 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.743841 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.743842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.743844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:56.743916 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.743918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.744590 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.744591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.745370 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.745372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.745913 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.745915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.745985 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.745988 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.746060 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.746095 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260925 19:00:56.739465 8464 raft_consensus.cc:2258] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:56.746464 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f538bfa97347eb90be232893a8fae5" candidate_uuid: "18048183b997408c812b70c748f2c8a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0310a7fbe57d49c7bd592dbeef606a21" is_pre_election: true
I20260925 19:00:56.746578 8231 raft_consensus.cc:2454] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18048183b997408c812b70c748f2c8a9 in term 0.
I20260925 19:00:56.746672 8464 ts_tablet_manager.cc:1905] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:56.746788 8025 leader_election.cc:302] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0310a7fbe57d49c7bd592dbeef606a21, 18048183b997408c812b70c748f2c8a9; no voters:
I20260925 19:00:56.746879 8450 raft_consensus.cc:2781] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:56.746904 8450 raft_consensus.cc:482] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:56.746922 8450 raft_consensus.cc:3037] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:56.747314 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.743522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40857 (local address 127.0.98.1:40645)
0925 19:00:56.743647 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.743657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.743693 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.743911 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.743912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.743913 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:56.743965 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.743967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.744813 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.744814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.745873 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.745875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.747084 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.747085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.747162 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.747165 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.747219 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.747252 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":52,"threads_started":1}
I20260925 19:00:56.747615 8450 raft_consensus.cc:504] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.747671 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.743760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40645 (local address 127.0.98.3:40857)
0925 19:00:56.743794 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.743796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.743802 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:00:56.743822 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.743823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.743830 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.743875 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:56.744033 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.744723 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.746034 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.746600 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.746666 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.746676 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.747574 (+ 898us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.747611 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:56.747730 8450 leader_election.cc:288] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [CANDIDATE]: Term 1 election: Requested vote from peers 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951), 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857)
I20260925 19:00:56.747836 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f538bfa97347eb90be232893a8fae5" candidate_uuid: "18048183b997408c812b70c748f2c8a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b6a652fa314ab6a553fd3b14d153fb" is_pre_election: true
I20260925 19:00:56.747895 8371 raft_consensus.cc:2454] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18048183b997408c812b70c748f2c8a9 in term 0.
I20260925 19:00:56.747903 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f538bfa97347eb90be232893a8fae5" candidate_uuid: "18048183b997408c812b70c748f2c8a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0310a7fbe57d49c7bd592dbeef606a21"
I20260925 19:00:56.747963 8231 raft_consensus.cc:3037] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:56.748291 8464 ts_tablet_manager.cc:1918] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:56.748375 8464 log.cc:1196] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/42e38924b7d94301956c675a4084ec52
I20260925 19:00:56.748631 8464 ts_tablet_manager.cc:1939] T 42e38924b7d94301956c675a4084ec52 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting consensus metadata
I20260925 19:00:56.748726 8231 raft_consensus.cc:2454] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18048183b997408c812b70c748f2c8a9 in term 1.
I20260925 19:00:56.749059 7918 catalog_manager.cc:4841] TS 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857): tablet 42e38924b7d94301956c675a4084ec52 (table test-table [id=401389ba90d64a5d8cd5c5f30c1dd23a]) successfully deleted
I20260925 19:00:56.749102 8025 leader_election.cc:302] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0310a7fbe57d49c7bd592dbeef606a21, 18048183b997408c812b70c748f2c8a9; no voters:
I20260925 19:00:56.749097 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f538bfa97347eb90be232893a8fae5" candidate_uuid: "18048183b997408c812b70c748f2c8a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b6a652fa314ab6a553fd3b14d153fb"
I20260925 19:00:56.749155 8371 raft_consensus.cc:3037] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:56.749169 8450 raft_consensus.cc:2781] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:56.749271 8450 raft_consensus.cc:686] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 1 LEADER]: Becoming Leader. State: Replica: 18048183b997408c812b70c748f2c8a9, State: Running, Role: LEADER
I20260925 19:00:56.749321 8450 consensus_queue.cc:237] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.749787 7934 catalog_manager.cc:5495] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18048183b997408c812b70c748f2c8a9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "18048183b997408c812b70c748f2c8a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } health_report { overall_health: UNKNOWN } } }
I20260925 19:00:56.749862 8371 raft_consensus.cc:2454] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18048183b997408c812b70c748f2c8a9 in term 1.
Sep 25 19:00:56 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:56.778793 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.770913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38400 (local address 127.0.98.62:40119)
0925 19:00:56.770958 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.770959 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.771066 (+ 107us) server_negotiation.cc:407] Connection header received
0925 19:00:56.771140 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.771141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.771150 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.771214 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:56.771758 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.772286 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.772928 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.773184 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.773244 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.775827 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:56.775828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:56.775829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:56.775837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:56.777447 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.777836 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.777837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.777837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.777913 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:56.778124 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:56.778125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:56.778125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:56.778320 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:56.778365 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.778588 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.778664 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:00:56.779085 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.770872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.0.1:38400)
0925 19:00:56.771009 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.771018 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.771090 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.771217 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.771218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.771588 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:56.771667 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.771672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.772361 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.772362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.772838 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.772839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.773293 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.773294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.773348 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.773906 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:56.773911 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:56.775735 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:56.777532 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.777539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.777541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.777761 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.777991 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:56.777992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:56.777992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:56.778055 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:56.778409 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:56.778416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:56.778478 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.778905 (+ 427us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.778997 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260925 19:00:56.781097 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38400:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 19:00:56.781329 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:56.782707 8071 tablet_service.cc:1467] Processing CreateTablet for tablet fa7c73048ebd41fda2ceb5ea9eb289ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=816e0d52bad7489faf5df887d9dbe95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:00:56.782807 8351 tablet_service.cc:1467] Processing CreateTablet for tablet fa7c73048ebd41fda2ceb5ea9eb289ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=816e0d52bad7489faf5df887d9dbe95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:00:56.782869 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7c73048ebd41fda2ceb5ea9eb289ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:56.782877 8211 tablet_service.cc:1467] Processing CreateTablet for tablet fa7c73048ebd41fda2ceb5ea9eb289ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=816e0d52bad7489faf5df887d9dbe95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 19:00:56.782929 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7c73048ebd41fda2ceb5ea9eb289ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:56.783001 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7c73048ebd41fda2ceb5ea9eb289ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:56.784348 8447 tablet_bootstrap.cc:492] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: Bootstrap starting.
I20260925 19:00:56.784348 8443 tablet_bootstrap.cc:492] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: Bootstrap starting.
I20260925 19:00:56.784729 8443 tablet_bootstrap.cc:654] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:56.784729 8447 tablet_bootstrap.cc:654] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:56.785071 8442 tablet_bootstrap.cc:492] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: Bootstrap starting.
I20260925 19:00:56.785497 8442 tablet_bootstrap.cc:654] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:56.786098 8442 tablet_bootstrap.cc:492] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: No bootstrap required, opened a new log
I20260925 19:00:56.786144 8442 ts_tablet_manager.cc:1397] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:56.786276 8442 raft_consensus.cc:348] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.786365 8442 raft_consensus.cc:374] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:56.786383 8442 raft_consensus.cc:729] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b6a652fa314ab6a553fd3b14d153fb, State: Initialized, Role: FOLLOWER
I20260925 19:00:56.786434 8442 consensus_queue.cc:260] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.786513 8442 ts_tablet_manager.cc:1428] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:56.786576 8443 tablet_bootstrap.cc:492] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: No bootstrap required, opened a new log
I20260925 19:00:56.786626 8443 ts_tablet_manager.cc:1397] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:56.786775 8443 raft_consensus.cc:348] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.786849 8443 raft_consensus.cc:374] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:56.786872 8443 raft_consensus.cc:729] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0310a7fbe57d49c7bd592dbeef606a21, State: Initialized, Role: FOLLOWER
I20260925 19:00:56.786913 8443 consensus_queue.cc:260] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.787029 8443 ts_tablet_manager.cc:1428] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:56.787153 8448 tablet_replica.cc:439] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } }
I20260925 19:00:56.787264 8447 tablet_bootstrap.cc:492] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: No bootstrap required, opened a new log
I20260925 19:00:56.787256 8448 tablet_replica.cc:442] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:56.787302 8447 ts_tablet_manager.cc:1397] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:00:56.787439 8447 raft_consensus.cc:348] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.787525 8447 raft_consensus.cc:374] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:56.787549 8447 raft_consensus.cc:729] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18048183b997408c812b70c748f2c8a9, State: Initialized, Role: FOLLOWER
I20260925 19:00:56.787038 8452 tablet_replica.cc:439] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } }
I20260925 19:00:56.787590 8447 consensus_queue.cc:260] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.787627 8452 tablet_replica.cc:442] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:56.787680 8447 ts_tablet_manager.cc:1428] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 19:00:56.787781 8450 tablet_replica.cc:439] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } }
I20260925 19:00:56.787846 8450 tablet_replica.cc:442] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 19:00:56.821445 8138 tablet.cc:2369] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:00:56.866648 8418 tablet.cc:2369] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:00:56.945407 8450 raft_consensus.cc:482] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:56.945504 8450 raft_consensus.cc:504] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.945680 8450 leader_election.cc:288] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951), 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857)
I20260925 19:00:56.945868 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7c73048ebd41fda2ceb5ea9eb289ba" candidate_uuid: "18048183b997408c812b70c748f2c8a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0310a7fbe57d49c7bd592dbeef606a21" is_pre_election: true
I20260925 19:00:56.945994 8231 raft_consensus.cc:2454] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18048183b997408c812b70c748f2c8a9 in term 0.
I20260925 19:00:56.945924 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7c73048ebd41fda2ceb5ea9eb289ba" candidate_uuid: "18048183b997408c812b70c748f2c8a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b6a652fa314ab6a553fd3b14d153fb" is_pre_election: true
I20260925 19:00:56.946049 8371 raft_consensus.cc:2454] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18048183b997408c812b70c748f2c8a9 in term 0.
I20260925 19:00:56.946211 8022 leader_election.cc:302] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18048183b997408c812b70c748f2c8a9, 39b6a652fa314ab6a553fd3b14d153fb; no voters:
I20260925 19:00:56.946326 8450 raft_consensus.cc:2781] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:56.946380 8450 raft_consensus.cc:482] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:56.946403 8450 raft_consensus.cc:3037] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:56.947034 8450 raft_consensus.cc:504] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.947185 8450 leader_election.cc:288] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [CANDIDATE]: Term 1 election: Requested vote from peers 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951), 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857)
I20260925 19:00:56.947327 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7c73048ebd41fda2ceb5ea9eb289ba" candidate_uuid: "18048183b997408c812b70c748f2c8a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0310a7fbe57d49c7bd592dbeef606a21"
I20260925 19:00:56.947422 8231 raft_consensus.cc:3037] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:56.947507 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7c73048ebd41fda2ceb5ea9eb289ba" candidate_uuid: "18048183b997408c812b70c748f2c8a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b6a652fa314ab6a553fd3b14d153fb"
I20260925 19:00:56.947576 8371 raft_consensus.cc:3037] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:56.948076 8231 raft_consensus.cc:2454] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18048183b997408c812b70c748f2c8a9 in term 1.
I20260925 19:00:56.948138 8371 raft_consensus.cc:2454] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18048183b997408c812b70c748f2c8a9 in term 1.
I20260925 19:00:56.948288 8025 leader_election.cc:302] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0310a7fbe57d49c7bd592dbeef606a21, 18048183b997408c812b70c748f2c8a9; no voters:
I20260925 19:00:56.948424 8450 raft_consensus.cc:2781] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:56.948473 8450 raft_consensus.cc:686] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [term 1 LEADER]: Becoming Leader. State: Replica: 18048183b997408c812b70c748f2c8a9, State: Running, Role: LEADER
I20260925 19:00:56.948503 8450 consensus_queue.cc:237] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:56.948801 8471 tablet_replica.cc:439] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: TxnStatusTablet state changed. Reason: New leader 18048183b997408c812b70c748f2c8a9. Latest consensus state: current_term: 1 leader_uuid: "18048183b997408c812b70c748f2c8a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } }
I20260925 19:00:56.948897 8471 tablet_replica.cc:442] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:00:56.949054 8483 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 19:00:56.949142 8483 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 19:00:56.949090 7934 catalog_manager.cc:5495] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18048183b997408c812b70c748f2c8a9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "18048183b997408c812b70c748f2c8a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } health_report { overall_health: UNKNOWN } } }
I20260925 19:00:56.967804 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.965203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.0.1:37638)
0925 19:00:56.965276 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.965288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.965341 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.965486 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.965486 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.965487 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:56.965528 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.965530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.966390 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.966391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.966935 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.966936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.967376 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.967377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.967417 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.967431 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:56.967675 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:56.967681 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.967724 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.967757 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 19:00:56.967826 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.965282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.0.98.1:34875)
0925 19:00:56.965324 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.965327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.965335 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:56.965398 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.965399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.965414 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.965478 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:56.965568 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.966321 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.967011 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.967305 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.967383 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.967504 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:56.967616 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:56.967682 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.967728 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.967747 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:00:56.968811 8371 raft_consensus.cc:1261] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Refusing update from remote peer 18048183b997408c812b70c748f2c8a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:56.968890 8231 raft_consensus.cc:1261] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Refusing update from remote peer 18048183b997408c812b70c748f2c8a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:56.969170 8471 consensus_queue.cc:1035] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:56.969297 8450 consensus_queue.cc:1035] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:56.969772 8448 tablet_replica.cc:439] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: TxnStatusTablet state changed. Reason: New leader 18048183b997408c812b70c748f2c8a9. Latest consensus state: current_term: 1 leader_uuid: "18048183b997408c812b70c748f2c8a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } }
I20260925 19:00:56.969887 8448 tablet_replica.cc:442] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:56.969918 8452 tablet_replica.cc:439] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: TxnStatusTablet state changed. Reason: New leader 18048183b997408c812b70c748f2c8a9. Latest consensus state: current_term: 1 leader_uuid: "18048183b997408c812b70c748f2c8a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } }
I20260925 19:00:56.970005 8452 tablet_replica.cc:442] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:56.970544 8471 tablet_replica.cc:439] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18048183b997408c812b70c748f2c8a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } }
I20260925 19:00:56.970633 8471 tablet_replica.cc:442] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:00:56.971086 8448 tablet_replica.cc:439] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18048183b997408c812b70c748f2c8a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } }
I20260925 19:00:56.971122 8457 mvcc.cc:205] Tried to move back new op lower bound from 7333326262142459904 to 7333326262061617152. Current Snapshot: MvccSnapshot[applied={T|T < 7333326262142459904 or (T in {7333326262142459904})}]
I20260925 19:00:56.971163 8448 tablet_replica.cc:442] T fa7c73048ebd41fda2ceb5ea9eb289ba P 39b6a652fa314ab6a553fd3b14d153fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:56.971306 8450 tablet_replica.cc:439] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18048183b997408c812b70c748f2c8a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } }
I20260925 19:00:56.971387 8450 tablet_replica.cc:442] T fa7c73048ebd41fda2ceb5ea9eb289ba P 18048183b997408c812b70c748f2c8a9: This TxnStatusTablet replica's current role is: LEADER
I20260925 19:00:56.971799 8452 tablet_replica.cc:439] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18048183b997408c812b70c748f2c8a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } } }
I20260925 19:00:56.971908 8452 tablet_replica.cc:442] T fa7c73048ebd41fda2ceb5ea9eb289ba P 0310a7fbe57d49c7bd592dbeef606a21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 19:00:56.974716 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.972463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.0.98.1:34875)
0925 19:00:56.972493 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.972495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.972520 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:00:56.972587 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.972588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.972604 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.972665 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:56.972802 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.973243 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.974056 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.974241 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.974352 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.974498 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:56.974525 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:56.974586 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.974640 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.974655 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 19:00:56.979939 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.977385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54317 (local address 127.0.98.1:34875)
0925 19:00:56.977432 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:56.977434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:56.977558 (+ 124us) server_negotiation.cc:407] Connection header received
0925 19:00:56.977601 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:56.977601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:56.977609 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:56.977664 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:56.977771 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.978174 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.979350 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.979525 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.979587 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.979763 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:56.979785 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:56.979822 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:56.979865 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 19:00:56.979877 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 19:00:56.980044 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:56.977339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.98.1:54317)
0925 19:00:56.977492 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:56.977502 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:56.977556 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:56.977666 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:56.977667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:56.977668 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:56.977719 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.977720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.978244 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.978245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.979208 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:56.979211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:56.979605 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:56.979606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:56.979672 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:56.979686 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:56.979823 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:56.979825 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:00:56.979866 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 19:00:56.979902 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260925 19:00:56.981985 8231 raft_consensus.cc:1261] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Refusing update from remote peer 18048183b997408c812b70c748f2c8a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:56.982046 8371 raft_consensus.cc:1261] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Refusing update from remote peer 18048183b997408c812b70c748f2c8a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:56.982164 8471 consensus_queue.cc:1035] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:56.982275 8450 consensus_queue.cc:1035] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:56.992825 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:56.992921 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:56.993520 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2: Sending DeleteTablet for 3 replicas of tablet 62f538bfa97347eb90be232893a8fae5
I20260925 19:00:56.993775 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f538bfa97347eb90be232893a8fae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42654
I20260925 19:00:56.993780 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f538bfa97347eb90be232893a8fae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52874
I20260925 19:00:56.993891 8466 tablet_replica.cc:330] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21: stopping tablet replica
I20260925 19:00:56.993891 8464 tablet_replica.cc:330] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: stopping tablet replica
I20260925 19:00:56.993906 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f538bfa97347eb90be232893a8fae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37626
I20260925 19:00:56.993956 8466 raft_consensus.cc:2229] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:56.993963 8464 raft_consensus.cc:2229] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:56.994024 8464 raft_consensus.cc:2258] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:56.994027 8466 raft_consensus.cc:2258] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:56.994107 8465 tablet_replica.cc:330] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: stopping tablet replica
I20260925 19:00:56.994159 8465 raft_consensus.cc:2229] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:56.994259 8465 raft_consensus.cc:2258] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:56.994354 8464 ts_tablet_manager.cc:1905] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:56.994621 8465 ts_tablet_manager.cc:1905] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:56.996047 8464 ts_tablet_manager.cc:1918] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:56.996104 8464 log.cc:1196] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/62f538bfa97347eb90be232893a8fae5
I20260925 19:00:56.996119 8465 ts_tablet_manager.cc:1918] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:56.996162 8465 log.cc:1196] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/62f538bfa97347eb90be232893a8fae5
I20260925 19:00:56.996313 8464 ts_tablet_manager.cc:1939] T 62f538bfa97347eb90be232893a8fae5 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting consensus metadata
I20260925 19:00:56.996411 8465 ts_tablet_manager.cc:1939] T 62f538bfa97347eb90be232893a8fae5 P 18048183b997408c812b70c748f2c8a9: Deleting consensus metadata
I20260925 19:00:56.996626 7918 catalog_manager.cc:4841] TS 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857): tablet 62f538bfa97347eb90be232893a8fae5 (table test-table [id=5c8447ee39aa4514b1006a3c56df9e30]) successfully deleted
I20260925 19:00:56.996760 7918 catalog_manager.cc:4841] TS 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875): tablet 62f538bfa97347eb90be232893a8fae5 (table test-table [id=5c8447ee39aa4514b1006a3c56df9e30]) successfully deleted
I20260925 19:00:56.996965 8466 ts_tablet_manager.cc:1905] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:56.998407 8466 ts_tablet_manager.cc:1918] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:56.998462 8466 log.cc:1196] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/62f538bfa97347eb90be232893a8fae5
I20260925 19:00:56.998670 8466 ts_tablet_manager.cc:1939] T 62f538bfa97347eb90be232893a8fae5 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting consensus metadata
I20260925 19:00:56.999071 7921 catalog_manager.cc:4841] TS 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951): tablet 62f538bfa97347eb90be232893a8fae5 (table test-table [id=5c8447ee39aa4514b1006a3c56df9e30]) successfully deleted
Sep 25 19:00:57 dist-test-slave-49bz krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 19:00:57 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:00:57 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 19:00:57.015535 8003 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:00:57.009868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.0.98.62:40119)
0925 19:00:57.009954 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:57.009956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:57.010075 (+ 119us) server_negotiation.cc:407] Connection header received
0925 19:00:57.010122 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:57.010122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:57.010132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:57.010183 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:57.010780 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.011549 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.012275 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.012579 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.012651 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.015473 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38416: BlockingRecv error: recv got EOF from 127.0.0.1:38416 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 19:00:57 dist-test-slave-49bz krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 19:00:57.022782 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.016285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.0.98.62:40119)
0925 19:00:57.016315 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:57.016317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:57.016398 (+ 81us) server_negotiation.cc:407] Connection header received
0925 19:00:57.016449 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:57.016449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:57.016458 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:57.016503 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:57.016916 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.017347 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.018052 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.018283 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.018338 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.020409 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:57.020411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:57.020411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:57.020418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:57.021742 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:57.022060 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:57.022061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:57.022062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:57.022116 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:57.022350 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:57.022350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:57.022351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:57.022451 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:57.022548 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:57.022631 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 19:00:57.022698 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 19:00:57.024111 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:00:57.024250 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:57.025484 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 19af0b8753ec460ba185f6fbc94404c7 (DEFAULT_TABLE table=test-table [id=9f6b765671264feca59c401066ce60ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:57.025542 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 19af0b8753ec460ba185f6fbc94404c7 (DEFAULT_TABLE table=test-table [id=9f6b765671264feca59c401066ce60ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:57.025556 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 19af0b8753ec460ba185f6fbc94404c7 (DEFAULT_TABLE table=test-table [id=9f6b765671264feca59c401066ce60ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:57.025600 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19af0b8753ec460ba185f6fbc94404c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:57.025646 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19af0b8753ec460ba185f6fbc94404c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:57.025648 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19af0b8753ec460ba185f6fbc94404c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:57.026823 8442 tablet_bootstrap.cc:492] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb: Bootstrap starting.
I20260925 19:00:57.026965 8443 tablet_bootstrap.cc:492] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21: Bootstrap starting.
I20260925 19:00:57.027331 8443 tablet_bootstrap.cc:654] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:57.027331 8442 tablet_bootstrap.cc:654] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:57.027619 8447 tablet_bootstrap.cc:492] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9: Bootstrap starting.
I20260925 19:00:57.027866 8443 tablet_bootstrap.cc:492] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21: No bootstrap required, opened a new log
I20260925 19:00:57.027908 8443 ts_tablet_manager.cc:1397] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:57.027992 8447 tablet_bootstrap.cc:654] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:57.028018 8443 raft_consensus.cc:348] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.028105 8443 raft_consensus.cc:374] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:57.028128 8443 raft_consensus.cc:729] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0310a7fbe57d49c7bd592dbeef606a21, State: Initialized, Role: FOLLOWER
I20260925 19:00:57.028172 8443 consensus_queue.cc:260] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.028280 8443 ts_tablet_manager.cc:1428] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:57.028621 8442 tablet_bootstrap.cc:492] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb: No bootstrap required, opened a new log
I20260925 19:00:57.028669 8442 ts_tablet_manager.cc:1397] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:00:57.028810 8442 raft_consensus.cc:348] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.028877 8442 raft_consensus.cc:374] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:57.028898 8442 raft_consensus.cc:729] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b6a652fa314ab6a553fd3b14d153fb, State: Initialized, Role: FOLLOWER
I20260925 19:00:57.028930 8442 consensus_queue.cc:260] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.029050 8442 ts_tablet_manager.cc:1428] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:57.029048 8447 tablet_bootstrap.cc:492] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9: No bootstrap required, opened a new log
I20260925 19:00:57.029089 8447 ts_tablet_manager.cc:1397] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:00:57.029218 8447 raft_consensus.cc:348] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.029281 8447 raft_consensus.cc:374] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:57.029313 8447 raft_consensus.cc:729] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18048183b997408c812b70c748f2c8a9, State: Initialized, Role: FOLLOWER
I20260925 19:00:57.029357 8447 consensus_queue.cc:260] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.029436 8447 ts_tablet_manager.cc:1428] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:57.140033 8448 raft_consensus.cc:482] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:57.140136 8448 raft_consensus.cc:504] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.140501 8448 leader_election.cc:288] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875), 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951)
I20260925 19:00:57.143365 8452 raft_consensus.cc:482] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:57.143441 8452 raft_consensus.cc:504] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.143565 8452 leader_election.cc:288] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875), 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857)
I20260925 19:00:57.143824 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19af0b8753ec460ba185f6fbc94404c7" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b6a652fa314ab6a553fd3b14d153fb" is_pre_election: true
I20260925 19:00:57.143903 8371 raft_consensus.cc:2454] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0310a7fbe57d49c7bd592dbeef606a21 in term 0.
I20260925 19:00:57.144065 8162 leader_election.cc:302] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0310a7fbe57d49c7bd592dbeef606a21, 39b6a652fa314ab6a553fd3b14d153fb; no voters:
I20260925 19:00:57.144052 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19af0b8753ec460ba185f6fbc94404c7" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18048183b997408c812b70c748f2c8a9" is_pre_election: true
I20260925 19:00:57.144136 8091 raft_consensus.cc:2454] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0310a7fbe57d49c7bd592dbeef606a21 in term 0.
I20260925 19:00:57.144150 8452 raft_consensus.cc:2781] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:57.144179 8452 raft_consensus.cc:482] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:57.144203 8452 raft_consensus.cc:3037] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:57.144486 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.140822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40651 (local address 127.0.98.2:35951)
0925 19:00:57.140866 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:57.140868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:57.140922 (+ 54us) server_negotiation.cc:407] Connection header received
0925 19:00:57.141093 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:57.141094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:57.141105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:57.141169 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:57.141388 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.142155 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.143570 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.144097 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.144145 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.144155 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:57.144381 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 19:00:57.144419 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:00:57.144680 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.140669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35951 (local address 127.0.98.3:40651)
0925 19:00:57.140831 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:57.140844 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:57.140982 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:57.141210 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:57.141211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:57.141213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:57.141280 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:57.141282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.142237 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.142238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:57.143458 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:57.143461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.144217 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.144217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:57.144296 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.144299 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:57.144554 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 19:00:57.144600 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":26,"thread_start_us":54,"threads_started":1}
I20260925 19:00:57.145246 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.140730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43743 (local address 127.0.98.1:34875)
0925 19:00:57.140781 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:57.140783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:57.140921 (+ 138us) server_negotiation.cc:407] Connection header received
0925 19:00:57.141044 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:57.141045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:57.141055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:57.141144 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:00:57.141329 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.142149 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.143267 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.143874 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.143944 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.143957 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:57.145154 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0925 19:00:57.145187 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 19:00:57.145247 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.140681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.98.3:43743)
0925 19:00:57.140831 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:57.140843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:57.140953 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:57.141160 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:57.141161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:57.141164 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:00:57.141240 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:57.141242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.142286 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.142288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:57.143165 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:57.143167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.144999 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.145000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:57.145070 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.145073 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:57.145129 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 19:00:57.145173 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260925 19:00:57.145437 8452 raft_consensus.cc:504] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.145542 8452 leader_election.cc:288] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 election: Requested vote from peers 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875), 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857)
I20260925 19:00:57.145792 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19af0b8753ec460ba185f6fbc94404c7" candidate_uuid: "39b6a652fa314ab6a553fd3b14d153fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18048183b997408c812b70c748f2c8a9" is_pre_election: true
I20260925 19:00:57.145828 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19af0b8753ec460ba185f6fbc94404c7" candidate_uuid: "39b6a652fa314ab6a553fd3b14d153fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0310a7fbe57d49c7bd592dbeef606a21" is_pre_election: true
I20260925 19:00:57.145825 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19af0b8753ec460ba185f6fbc94404c7" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18048183b997408c812b70c748f2c8a9"
I20260925 19:00:57.145872 8091 raft_consensus.cc:2454] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b6a652fa314ab6a553fd3b14d153fb in term 0.
I20260925 19:00:57.145910 8231 raft_consensus.cc:2379] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39b6a652fa314ab6a553fd3b14d153fb in current term 1: Already voted for candidate 0310a7fbe57d49c7bd592dbeef606a21 in this term.
I20260925 19:00:57.146070 8302 leader_election.cc:302] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18048183b997408c812b70c748f2c8a9, 39b6a652fa314ab6a553fd3b14d153fb; no voters:
I20260925 19:00:57.146189 8448 raft_consensus.cc:2781] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:57.146237 8448 raft_consensus.cc:482] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:57.146307 8448 raft_consensus.cc:3037] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:57.146389 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19af0b8753ec460ba185f6fbc94404c7" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b6a652fa314ab6a553fd3b14d153fb"
I20260925 19:00:57.146983 8448 raft_consensus.cc:504] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.147109 8371 raft_consensus.cc:2379] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0310a7fbe57d49c7bd592dbeef606a21 in current term 1: Already voted for candidate 39b6a652fa314ab6a553fd3b14d153fb in this term.
I20260925 19:00:57.147130 8448 leader_election.cc:288] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [CANDIDATE]: Term 1 election: Requested vote from peers 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875), 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951)
I20260925 19:00:57.147264 8162 leader_election.cc:302] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0310a7fbe57d49c7bd592dbeef606a21; no voters: 18048183b997408c812b70c748f2c8a9, 39b6a652fa314ab6a553fd3b14d153fb
I20260925 19:00:57.147399 8452 raft_consensus.cc:2726] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 19:00:57.147408 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19af0b8753ec460ba185f6fbc94404c7" candidate_uuid: "39b6a652fa314ab6a553fd3b14d153fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18048183b997408c812b70c748f2c8a9"
I20260925 19:00:57.147408 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19af0b8753ec460ba185f6fbc94404c7" candidate_uuid: "39b6a652fa314ab6a553fd3b14d153fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0310a7fbe57d49c7bd592dbeef606a21"
I20260925 19:00:57.147467 8090 raft_consensus.cc:3037] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:57.147504 8231 raft_consensus.cc:2379] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39b6a652fa314ab6a553fd3b14d153fb in current term 1: Already voted for candidate 0310a7fbe57d49c7bd592dbeef606a21 in this term.
I20260925 19:00:57.148078 8090 raft_consensus.cc:2454] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b6a652fa314ab6a553fd3b14d153fb in term 1.
I20260925 19:00:57.148231 8302 leader_election.cc:302] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18048183b997408c812b70c748f2c8a9, 39b6a652fa314ab6a553fd3b14d153fb; no voters: 0310a7fbe57d49c7bd592dbeef606a21
I20260925 19:00:57.148331 8448 raft_consensus.cc:2781] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:57.148471 8448 raft_consensus.cc:686] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 LEADER]: Becoming Leader. State: Replica: 39b6a652fa314ab6a553fd3b14d153fb, State: Running, Role: LEADER
I20260925 19:00:57.148540 8448 consensus_queue.cc:237] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.149284 7932 catalog_manager.cc:5495] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "39b6a652fa314ab6a553fd3b14d153fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } health_report { overall_health: HEALTHY } } }
I20260925 19:00:57.174783 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.172461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.0.98.3:40857)
0925 19:00:57.172516 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:57.172518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:57.172525 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:00:57.172562 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:57.172562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:57.172578 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:57.172654 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:57.172762 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.173234 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.173990 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.174174 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.174249 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.174463 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:57.174553 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:57.174628 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:57.174668 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 19:00:57.174685 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:00:57.175727 8090 raft_consensus.cc:1261] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Refusing update from remote peer 39b6a652fa314ab6a553fd3b14d153fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:57.175783 8231 raft_consensus.cc:1261] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Refusing update from remote peer 39b6a652fa314ab6a553fd3b14d153fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:57.175971 8448 consensus_queue.cc:1035] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:57.176139 8512 consensus_queue.cc:1035] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:57.179133 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:57.179236 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:57.179800 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2: Sending DeleteTablet for 3 replicas of tablet 19af0b8753ec460ba185f6fbc94404c7
I20260925 19:00:57.180042 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 19af0b8753ec460ba185f6fbc94404c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52874
I20260925 19:00:57.180122 8466 tablet_replica.cc:330] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21: stopping tablet replica
I20260925 19:00:57.180168 8466 raft_consensus.cc:2229] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:57.180212 8466 raft_consensus.cc:2258] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:57.180395 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 19:00:57.180486 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 19af0b8753ec460ba185f6fbc94404c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37626
W20260925 19:00:57.180529 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:00:57.180567 8465 tablet_replica.cc:330] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9: stopping tablet replica
I20260925 19:00:57.180608 8465 raft_consensus.cc:2229] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:57.180641 8465 raft_consensus.cc:2258] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:57.180742 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 19af0b8753ec460ba185f6fbc94404c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42654
I20260925 19:00:57.180807 8464 tablet_replica.cc:330] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb: stopping tablet replica
I20260925 19:00:57.180848 8464 raft_consensus.cc:2229] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:57.180913 8464 raft_consensus.cc:2258] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:57.181175 8464 ts_tablet_manager.cc:1905] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:57.181670 8211 tablet_service.cc:1467] Processing CreateTablet for tablet dec4bc7d536041bca04a180fd7350099 (DEFAULT_TABLE table=test-table [id=bef4e838b0934089b343e07c98e9860b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:57.181764 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec4bc7d536041bca04a180fd7350099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:57.182621 8464 ts_tablet_manager.cc:1918] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:57.182663 8464 log.cc:1196] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/19af0b8753ec460ba185f6fbc94404c7
I20260925 19:00:57.182854 8464 ts_tablet_manager.cc:1939] T 19af0b8753ec460ba185f6fbc94404c7 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting consensus metadata
I20260925 19:00:57.183120 8443 tablet_bootstrap.cc:492] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21: Bootstrap starting.
I20260925 19:00:57.183290 8465 ts_tablet_manager.cc:1905] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:57.183449 8443 tablet_bootstrap.cc:654] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:57.183828 7918 catalog_manager.cc:4841] TS 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857): tablet 19af0b8753ec460ba185f6fbc94404c7 (table test-table [id=9f6b765671264feca59c401066ce60ab]) successfully deleted
I20260925 19:00:57.183984 8070 tablet_service.cc:1467] Processing CreateTablet for tablet dec4bc7d536041bca04a180fd7350099 (DEFAULT_TABLE table=test-table [id=bef4e838b0934089b343e07c98e9860b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:57.184115 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec4bc7d536041bca04a180fd7350099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:57.184772 8465 ts_tablet_manager.cc:1918] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:57.184818 8465 log.cc:1196] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/19af0b8753ec460ba185f6fbc94404c7
I20260925 19:00:57.185017 8465 ts_tablet_manager.cc:1939] T 19af0b8753ec460ba185f6fbc94404c7 P 18048183b997408c812b70c748f2c8a9: Deleting consensus metadata
I20260925 19:00:57.185422 7918 catalog_manager.cc:4841] TS 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875): tablet 19af0b8753ec460ba185f6fbc94404c7 (table test-table [id=9f6b765671264feca59c401066ce60ab]) successfully deleted
I20260925 19:00:57.185484 8447 tablet_bootstrap.cc:492] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9: Bootstrap starting.
I20260925 19:00:57.185755 8447 tablet_bootstrap.cc:654] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:57.186012 8351 tablet_service.cc:1467] Processing CreateTablet for tablet dec4bc7d536041bca04a180fd7350099 (DEFAULT_TABLE table=test-table [id=bef4e838b0934089b343e07c98e9860b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:00:57.186095 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec4bc7d536041bca04a180fd7350099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:57.186481 8443 tablet_bootstrap.cc:492] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21: No bootstrap required, opened a new log
I20260925 19:00:57.186520 8443 ts_tablet_manager.cc:1397] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:00:57.186616 8443 raft_consensus.cc:348] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.186685 8443 raft_consensus.cc:374] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:57.186707 8443 raft_consensus.cc:729] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0310a7fbe57d49c7bd592dbeef606a21, State: Initialized, Role: FOLLOWER
I20260925 19:00:57.186751 8443 consensus_queue.cc:260] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.186846 8443 ts_tablet_manager.cc:1428] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:57.187330 8442 tablet_bootstrap.cc:492] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb: Bootstrap starting.
I20260925 19:00:57.187471 8466 ts_tablet_manager.cc:1905] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:57.187678 8442 tablet_bootstrap.cc:654] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:57.188249 8447 tablet_bootstrap.cc:492] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9: No bootstrap required, opened a new log
I20260925 19:00:57.188289 8447 ts_tablet_manager.cc:1397] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260925 19:00:57.188377 8447 raft_consensus.cc:348] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.188441 8447 raft_consensus.cc:374] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:57.188459 8447 raft_consensus.cc:729] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18048183b997408c812b70c748f2c8a9, State: Initialized, Role: FOLLOWER
I20260925 19:00:57.188495 8447 consensus_queue.cc:260] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.188567 8447 ts_tablet_manager.cc:1428] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:57.188822 8466 ts_tablet_manager.cc:1918] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:00:57.188863 8466 log.cc:1196] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/19af0b8753ec460ba185f6fbc94404c7
I20260925 19:00:57.189065 8466 ts_tablet_manager.cc:1939] T 19af0b8753ec460ba185f6fbc94404c7 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting consensus metadata
I20260925 19:00:57.189415 7921 catalog_manager.cc:4841] TS 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951): tablet 19af0b8753ec460ba185f6fbc94404c7 (table test-table [id=9f6b765671264feca59c401066ce60ab]) successfully deleted
I20260925 19:00:57.190145 8442 tablet_bootstrap.cc:492] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb: No bootstrap required, opened a new log
I20260925 19:00:57.190212 8442 ts_tablet_manager.cc:1397] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 19:00:57.190366 8442 raft_consensus.cc:348] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.190483 8442 raft_consensus.cc:374] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:57.190511 8442 raft_consensus.cc:729] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b6a652fa314ab6a553fd3b14d153fb, State: Initialized, Role: FOLLOWER
I20260925 19:00:57.190548 8442 consensus_queue.cc:260] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.190640 8442 ts_tablet_manager.cc:1428] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 19:00:57.232128 8452 raft_consensus.cc:482] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:00:57.232220 8452 raft_consensus.cc:504] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.232389 8452 leader_election.cc:288] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875), 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857)
I20260925 19:00:57.232544 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec4bc7d536041bca04a180fd7350099" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b6a652fa314ab6a553fd3b14d153fb" is_pre_election: true
I20260925 19:00:57.232545 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec4bc7d536041bca04a180fd7350099" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18048183b997408c812b70c748f2c8a9" is_pre_election: true
I20260925 19:00:57.232616 8371 raft_consensus.cc:2454] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0310a7fbe57d49c7bd592dbeef606a21 in term 0.
I20260925 19:00:57.232616 8090 raft_consensus.cc:2454] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0310a7fbe57d49c7bd592dbeef606a21 in term 0.
I20260925 19:00:57.232785 8162 leader_election.cc:302] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0310a7fbe57d49c7bd592dbeef606a21, 39b6a652fa314ab6a553fd3b14d153fb; no voters:
I20260925 19:00:57.232926 8452 raft_consensus.cc:2781] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:00:57.232975 8452 raft_consensus.cc:482] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:00:57.233002 8452 raft_consensus.cc:3037] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:57.233639 8452 raft_consensus.cc:504] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.233758 8452 leader_election.cc:288] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 election: Requested vote from peers 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875), 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857)
I20260925 19:00:57.234010 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec4bc7d536041bca04a180fd7350099" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18048183b997408c812b70c748f2c8a9"
I20260925 19:00:57.234036 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec4bc7d536041bca04a180fd7350099" candidate_uuid: "0310a7fbe57d49c7bd592dbeef606a21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b6a652fa314ab6a553fd3b14d153fb"
I20260925 19:00:57.234103 8090 raft_consensus.cc:3037] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:57.234110 8371 raft_consensus.cc:3037] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:57.234813 8090 raft_consensus.cc:2454] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0310a7fbe57d49c7bd592dbeef606a21 in term 1.
I20260925 19:00:57.234895 8371 raft_consensus.cc:2454] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0310a7fbe57d49c7bd592dbeef606a21 in term 1.
I20260925 19:00:57.234972 8162 leader_election.cc:302] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0310a7fbe57d49c7bd592dbeef606a21, 18048183b997408c812b70c748f2c8a9; no voters:
I20260925 19:00:57.235055 8452 raft_consensus.cc:2781] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:57.235152 8452 raft_consensus.cc:686] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 LEADER]: Becoming Leader. State: Replica: 0310a7fbe57d49c7bd592dbeef606a21, State: Running, Role: LEADER
I20260925 19:00:57.235204 8452 consensus_queue.cc:237] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } }
I20260925 19:00:57.235656 7934 catalog_manager.cc:5495] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0310a7fbe57d49c7bd592dbeef606a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0310a7fbe57d49c7bd592dbeef606a21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 } health_report { overall_health: UNKNOWN } } }
I20260925 19:00:57.253248 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.250871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.0.98.2:35951)
0925 19:00:57.250908 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:57.250909 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:57.250918 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:00:57.250986 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:57.250986 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:57.250996 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:57.251047 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:57.251141 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.251631 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.252382 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.252746 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.252826 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.253005 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:57.253038 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:57.253102 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:57.253154 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 19:00:57.253175 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:00:57.258224 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.255737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59411 (local address 127.0.98.1:34875)
0925 19:00:57.255789 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:57.255791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:57.255965 (+ 174us) server_negotiation.cc:407] Connection header received
0925 19:00:57.255986 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:57.255986 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:57.255993 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:57.256038 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:57.256168 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.256596 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.257640 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.257794 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.257837 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.258017 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:57.258038 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:57.258089 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:57.258164 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 19:00:57.258176 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:00:57.258301 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.255690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.98.2:59411)
0925 19:00:57.255869 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:57.255881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:57.255944 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:57.256058 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:57.256059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:57.256060 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:57.256102 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:57.256103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.256674 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.256675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:57.257545 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:57.257548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.257866 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.257867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:57.257924 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.257938 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:57.258107 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:57.258110 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:00:57.258160 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:00:57.258210 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260925 19:00:57.260380 8090 raft_consensus.cc:1261] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Refusing update from remote peer 0310a7fbe57d49c7bd592dbeef606a21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:57.260380 8371 raft_consensus.cc:1261] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Refusing update from remote peer 0310a7fbe57d49c7bd592dbeef606a21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:00:57.260666 8452 consensus_queue.cc:1035] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18048183b997408c812b70c748f2c8a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:57.260790 8521 consensus_queue.cc:1035] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b6a652fa314ab6a553fd3b14d153fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:00:57.267998 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.265560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52597 (local address 127.0.98.2:35951)
0925 19:00:57.265594 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:57.265595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:57.265603 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:00:57.265655 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:57.265656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:57.265665 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:57.265708 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:00:57.265845 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.266381 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.267295 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.267592 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.267658 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.267803 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:00:57.267832 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:00:57.267885 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:57.267939 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 19:00:57.267954 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 19:00:57.268054 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.265522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35951 (local address 127.0.98.1:52597)
0925 19:00:57.265562 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:57.265572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:57.265616 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:57.265746 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:57.265747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:57.265748 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 19:00:57.265783 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:57.265784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.266447 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.266448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:57.267213 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:57.267216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:57.267668 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:57.267669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:57.267724 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:57.267730 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 19:00:57.267890 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 19:00:57.267891 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:00:57.267938 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:00:57.267982 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 19:00:57.273909 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:57.274001 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:00:57.274653 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f60f73e2cfd84ed387837d19b23776f2: Sending DeleteTablet for 3 replicas of tablet dec4bc7d536041bca04a180fd7350099
I20260925 19:00:57.274899 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet dec4bc7d536041bca04a180fd7350099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37626
I20260925 19:00:57.274991 8465 tablet_replica.cc:330] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9: stopping tablet replica
I20260925 19:00:57.274973 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet dec4bc7d536041bca04a180fd7350099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52874
I20260925 19:00:57.275035 8465 raft_consensus.cc:2229] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:57.275060 8466 tablet_replica.cc:330] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21: stopping tablet replica
I20260925 19:00:57.275070 8465 raft_consensus.cc:2258] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:57.275108 8466 raft_consensus.cc:2229] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:00:57.275152 8466 raft_consensus.cc:2258] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:57.275134 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet dec4bc7d536041bca04a180fd7350099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:00:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42654
I20260925 19:00:57.275269 8464 tablet_replica.cc:330] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb: stopping tablet replica
I20260925 19:00:57.275311 8464 raft_consensus.cc:2229] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:00:57.275341 8464 raft_consensus.cc:2258] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:00:57.275635 8466 ts_tablet_manager.cc:1905] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:57.275635 8464 ts_tablet_manager.cc:1905] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:57.276961 8465 ts_tablet_manager.cc:1905] T dec4bc7d536041bca04a180fd7350099 P 18048183b997408c812b70c748f2c8a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:00:57.277061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 8006
I20260925 19:00:57.277184 8464 ts_tablet_manager.cc:1918] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:57.277230 8464 log.cc:1196] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/dec4bc7d536041bca04a180fd7350099
I20260925 19:00:57.277405 8464 ts_tablet_manager.cc:1939] T dec4bc7d536041bca04a180fd7350099 P 39b6a652fa314ab6a553fd3b14d153fb: Deleting consensus metadata
I20260925 19:00:57.278208 8466 ts_tablet_manager.cc:1918] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 19:00:57.278255 8466 log.cc:1196] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/dec4bc7d536041bca04a180fd7350099
I20260925 19:00:57.278491 8466 ts_tablet_manager.cc:1939] T dec4bc7d536041bca04a180fd7350099 P 0310a7fbe57d49c7bd592dbeef606a21: Deleting consensus metadata
I20260925 19:00:57.278914 7918 catalog_manager.cc:4841] TS 39b6a652fa314ab6a553fd3b14d153fb (127.0.98.3:40857): tablet dec4bc7d536041bca04a180fd7350099 (table test-table [id=bef4e838b0934089b343e07c98e9860b]) successfully deleted
I20260925 19:00:57.279443 7921 catalog_manager.cc:4841] TS 0310a7fbe57d49c7bd592dbeef606a21 (127.0.98.2:35951): tablet dec4bc7d536041bca04a180fd7350099 (table test-table [id=bef4e838b0934089b343e07c98e9860b]) successfully deleted
W20260925 19:00:57.284008 7918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34875 (error 108)
I20260925 19:00:57.284248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 8146
I20260925 19:00:57.285778 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:57.284376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.0.1:37658)
0925 19:00:57.285691 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:57.285739 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1298,"thread_start_us":37,"threads_started":1}
W20260925 19:00:57.286041 7918 catalog_manager.cc:4584] TS 18048183b997408c812b70c748f2c8a9 (127.0.98.1:34875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dec4bc7d536041bca04a180fd7350099: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34875: connect: Connection refused (error 111)
I20260925 19:00:57.290122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 8286
I20260925 19:00:57.295145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 7902
2026-09-25T19:00:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2401 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:57 dist-test-slave-49bz krb5kdc[8533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:57 dist-test-slave-49bz krb5kdc[8533](info): set up 2 sockets
Sep 25 19:00:57 dist-test-slave-49bz krb5kdc[8533](info): commencing operation
krb5kdc: starting...
W20260925 19:00:58.298462 392 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:58 dist-test-slave-49bz krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:00:58.408948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36933
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:58.486879 8546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:58.487123 8546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:58.487159 8546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:58.488286 8546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:58.488355 8546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:58.488379 8546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:58.488399 8546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:58.488428 8546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:58.489853 8546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:58.490099 8546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:58.490336 8546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:58.494973 8552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:58.495020 8551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:58.494980 8554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:58.495235 8546 server_base.cc:1034] running on GCE node
I20260925 19:00:58.495481 8546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:58.495769 8546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:58.496929 8546 hybrid_clock.cc:648] HybridClock initialized: now 1790362858496915 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T19:00:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1209 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:00:58 dist-test-slave-49bz krb5kdc[8559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:00:58 dist-test-slave-49bz krb5kdc[8559](info): set up 2 sockets
Sep 25 19:00:58 dist-test-slave-49bz krb5kdc[8559](info): commencing operation
krb5kdc: starting...
W20260925 19:00:59.514590 392 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:00:59 dist-test-slave-49bz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:00:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:00:59.621222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43835
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:59.694701 8572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:59.694883 8572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:59.694902 8572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:59.695907 8572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:00:59.695940 8572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:59.695952 8572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:59.695963 8572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:00:59.695974 8572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:00:59.697225 8572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:59.697381 8572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:59.697520 8572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:59.701936 8577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:59.701941 8580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:59.702019 8578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:59.702179 8572 server_base.cc:1034] running on GCE node
I20260925 19:00:59.702414 8572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:59.702664 8572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:59.703795 8572 hybrid_clock.cc:648] HybridClock initialized: now 1790362859703785 us; error 27 us; skew 500 ppm
Sep 25 19:00:59 dist-test-slave-49bz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:59.706135 8572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:00:59.706615 8572 webserver.cc:460] Webserver started at http://127.0.98.62:42265/ using document root <none> and password file <none>
I20260925 19:00:59.706825 8572 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:59.706887 8572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:59.707008 8572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:59.707927 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "d94d2154b58e46bfbc0ec938b771a248"
format_stamp: "Formatted at 2026-09-25 19:00:59 on dist-test-slave-49bz"
server_key: "0ae8a81442ab105b13ab92d456bd23f7"
server_key_iv: "d41379e1fb1408e2ddc76fe6b41fdb48"
server_key_version: "encryptionkey@0"
I20260925 19:00:59.708278 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "d94d2154b58e46bfbc0ec938b771a248"
format_stamp: "Formatted at 2026-09-25 19:00:59 on dist-test-slave-49bz"
server_key: "0ae8a81442ab105b13ab92d456bd23f7"
server_key_iv: "d41379e1fb1408e2ddc76fe6b41fdb48"
server_key_version: "encryptionkey@0"
I20260925 19:00:59.709532 8572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:00:59.710211 8587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:59.710400 8572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:59.710477 8572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "d94d2154b58e46bfbc0ec938b771a248"
format_stamp: "Formatted at 2026-09-25 19:00:59 on dist-test-slave-49bz"
server_key: "0ae8a81442ab105b13ab92d456bd23f7"
server_key_iv: "d41379e1fb1408e2ddc76fe6b41fdb48"
server_key_version: "encryptionkey@0"
I20260925 19:00:59.710534 8572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:59.734138 8572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:59.736090 8572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:59.736207 8572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:59.739717 8572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43835
I20260925 19:00:59.739745 8639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43835 every 8 connection(s)
I20260925 19:00:59.740216 8572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:00:59.740805 8640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:00:59.742936 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: Bootstrap starting.
I20260925 19:00:59.743515 8640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: Neither blocks nor log segments found. Creating new log.
I20260925 19:00:59.743796 8640 log.cc:824] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: Log is configured to *not* fsync() on all Append() calls
I20260925 19:00:59.744441 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: No bootstrap required, opened a new log
I20260925 19:00:59.745183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 8572
I20260925 19:00:59.745270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:00:59.745455 392 external_mini_cluster.cc:1428] Setting key 20c2823e68813a713981b8fe7c9709dd
I20260925 19:00:59.746946 8640 raft_consensus.cc:348] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260925 19:00:59.747114 8640 raft_consensus.cc:374] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:00:59.747152 8640 raft_consensus.cc:729] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94d2154b58e46bfbc0ec938b771a248, State: Initialized, Role: FOLLOWER
I20260925 19:00:59.747282 8640 consensus_queue.cc:260] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260925 19:00:59.747364 8640 raft_consensus.cc:388] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:00:59.747385 8640 raft_consensus.cc:482] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:00:59.747442 8640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:00:59.747946 8640 raft_consensus.cc:504] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260925 19:00:59.748041 8640 leader_election.cc:302] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d94d2154b58e46bfbc0ec938b771a248; no voters:
I20260925 19:00:59.748169 8640 leader_election.cc:288] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:00:59.748235 8645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:00:59.748371 8640 sys_catalog.cc:564] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:00:59.748409 8645 raft_consensus.cc:686] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 1 LEADER]: Becoming Leader. State: Replica: d94d2154b58e46bfbc0ec938b771a248, State: Running, Role: LEADER
I20260925 19:00:59.748596 8645 consensus_queue.cc:237] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260925 19:00:59.748975 8647 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d94d2154b58e46bfbc0ec938b771a248. Latest consensus state: current_term: 1 leader_uuid: "d94d2154b58e46bfbc0ec938b771a248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } } }
I20260925 19:00:59.749083 8647 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [sys.catalog]: This master's current role is: LEADER
I20260925 19:00:59.749507 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d94d2154b58e46bfbc0ec938b771a248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } } }
I20260925 19:00:59.749583 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [sys.catalog]: This master's current role is: LEADER
W20260925 19:00:59.750372 8660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:00:59.750762 8660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:00:59.750859 8654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:00:59.751057 8654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:00:59.752573 8654 catalog_manager.cc:1347] Generated new cluster ID: 100efdb8651b4e47a9b9e2b1c59a85cb
I20260925 19:00:59.752622 8654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:00:59 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:59.756507 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:59.745927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.0.98.62:43835)
0925 19:00:59.746150 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:59.746159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:59.746190 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:00:59.746917 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:59.746925 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:59.746981 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:59.747163 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:59.747885 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:59.749286 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:59.750457 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:59.750980 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:59.751101 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:59.753495 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:59.753499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:59.753499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:59.753517 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:59.755109 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:59.755458 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:59.755461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:59.755461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:59.755563 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:59.755758 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:59.755759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:59.755759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:59.755973 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:59.756151 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:59.756213 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:00:59.756339 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
I20260925 19:00:59.771157 8654 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:00:59.771840 8654 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:00:59.780403 8654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: Generated new TSK 0
I20260925 19:00:59.780560 8654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:00:59.804378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:00:59.881747 8666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:00:59.881932 8666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:00:59.881954 8666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:00:59.883134 8666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:00:59.883176 8666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:00:59.883203 8666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:00:59.884663 8666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:00:59.884872 8666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:00:59.885090 8666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:00:59.889662 8672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:00:59.889668 8671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:59.889846 8666 server_base.cc:1034] running on GCE node
W20260925 19:00:59.890005 8674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:00:59.890239 8666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:00:59.890471 8666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:00:59.891602 8666 hybrid_clock.cc:648] HybridClock initialized: now 1790362859891594 us; error 25 us; skew 500 ppm
Sep 25 19:00:59 dist-test-slave-49bz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:00:59.894004 8666 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:00:59.894471 8666 webserver.cc:460] Webserver started at http://127.0.98.1:33717/ using document root <none> and password file <none>
I20260925 19:00:59.894691 8666 fs_manager.cc:359] Metadata directory not provided
I20260925 19:00:59.894754 8666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:00:59.894892 8666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:00:59.895840 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "6b8305d23fe04c588049517e109829b4"
format_stamp: "Formatted at 2026-09-25 19:00:59 on dist-test-slave-49bz"
server_key: "99d8c1969ffb01aa032e227717e0576d"
server_key_iv: "81c9b8a0fbeda8cbfc1cd2925343dc2e"
server_key_version: "encryptionkey@0"
I20260925 19:00:59.896193 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6b8305d23fe04c588049517e109829b4"
format_stamp: "Formatted at 2026-09-25 19:00:59 on dist-test-slave-49bz"
server_key: "99d8c1969ffb01aa032e227717e0576d"
server_key_iv: "81c9b8a0fbeda8cbfc1cd2925343dc2e"
server_key_version: "encryptionkey@0"
I20260925 19:00:59.897416 8666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:00:59.898406 8681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:59.898562 8666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:00:59.898679 8666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "6b8305d23fe04c588049517e109829b4"
format_stamp: "Formatted at 2026-09-25 19:00:59 on dist-test-slave-49bz"
server_key: "99d8c1969ffb01aa032e227717e0576d"
server_key_iv: "81c9b8a0fbeda8cbfc1cd2925343dc2e"
server_key_version: "encryptionkey@0"
I20260925 19:00:59.898751 8666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:00:59.920105 8666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:00:59.922226 8666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:00:59.922389 8666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:00:59.922571 8666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:00:59.922823 8666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:00:59.922852 8666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:59.922873 8666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:00:59.922885 8666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:00:59.928316 8666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38333
I20260925 19:00:59.928400 8794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38333 every 8 connection(s)
I20260925 19:00:59.928692 8666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:00:59 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:00:59.937109 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:59.929398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47553 (local address 127.0.98.62:43835)
0925 19:00:59.929439 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:00:59.929441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:00:59.929635 (+ 194us) server_negotiation.cc:407] Connection header received
0925 19:00:59.930580 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:00:59.930581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:00:59.930591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:00:59.930643 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:00:59.931177 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:59.931677 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:59.932560 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:00:59.932747 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:00:59.932796 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:59.934843 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:00:59.934844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:00:59.934844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:00:59.934850 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:00:59.936066 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:59.936394 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:59.936395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:59.936395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:59.936441 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:00:59.936641 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:00:59.936641 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:00:59.936641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:00:59.936732 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:00:59.936778 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 19:00:59.936991 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 19:00:59.937051 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:00:59.937295 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:00:59.929381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:47553)
0925 19:00:59.929515 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 19:00:59.929548 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:00:59.930492 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:00:59.930690 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:00:59.930693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:00:59.930923 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:00:59.931109 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:59.931113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:59.931750 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:59.931751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:59.932489 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:00:59.932491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:00:59.932821 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:00:59.932822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:00:59.932911 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:00:59.933394 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:00:59.933402 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:00:59.934764 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:00:59.936135 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:59.936140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:59.936142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:59.936331 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:59.936501 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:00:59.936501 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:00:59.936502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:00:59.936588 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:00:59.936784 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:00:59.936787 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:00:59.936882 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 19:00:59.936958 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:00:59.937073 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260925 19:00:59.937858 8795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43835
I20260925 19:00:59.937963 8795 heartbeater.cc:464] Registering TS with master...
I20260925 19:00:59.938134 8795 heartbeater.cc:511] Master 127.0.98.62:43835 requested a full tablet report, sending...
I20260925 19:00:59.938539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 8666
I20260925 19:00:59.938656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:00:59.938691 8604 ts_manager.cc:194] Registered new tserver with Master: 6b8305d23fe04c588049517e109829b4 (127.0.98.1:38333)
I20260925 19:00:59.938844 392 external_mini_cluster.cc:1428] Setting key b3f2ebbcb5d12b802904085d3dca7d47
I20260925 19:00:59.939388 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47553
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:00:59.960600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:01:00.035219 8800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:00.035395 8800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:00.035418 8800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:00.036481 8800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:00.036517 8800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:00.036541 8800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:01:00.037971 8800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:00.038194 8800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:00.038425 8800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:01:00.042930 8800 server_base.cc:1034] running on GCE node
W20260925 19:01:00.042898 8805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:00.042887 8808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:00.042924 8806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:00.043318 8800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:00.043574 8800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:00.044730 8800 hybrid_clock.cc:648] HybridClock initialized: now 1790362860044707 us; error 39 us; skew 500 ppm
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:00.051141 8800 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:01:00.051460 8800 webserver.cc:460] Webserver started at http://127.0.98.2:45013/ using document root <none> and password file <none>
I20260925 19:01:00.051620 8800 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:00.051656 8800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:00.051729 8800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:00.052587 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "71bd901d441d4a7b89684e4f85049a21"
format_stamp: "Formatted at 2026-09-25 19:01:00 on dist-test-slave-49bz"
server_key: "57a19006acaae9015a126f6a4db4eb7d"
server_key_iv: "362caad54cccf2099c4f77cd1e306e6c"
server_key_version: "encryptionkey@0"
I20260925 19:01:00.052845 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "71bd901d441d4a7b89684e4f85049a21"
format_stamp: "Formatted at 2026-09-25 19:01:00 on dist-test-slave-49bz"
server_key: "57a19006acaae9015a126f6a4db4eb7d"
server_key_iv: "362caad54cccf2099c4f77cd1e306e6c"
server_key_version: "encryptionkey@0"
I20260925 19:01:00.054013 8800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:01:00.054749 8815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.054884 8800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.054956 8800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "71bd901d441d4a7b89684e4f85049a21"
format_stamp: "Formatted at 2026-09-25 19:01:00 on dist-test-slave-49bz"
server_key: "57a19006acaae9015a126f6a4db4eb7d"
server_key_iv: "362caad54cccf2099c4f77cd1e306e6c"
server_key_version: "encryptionkey@0"
I20260925 19:01:00.055003 8800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:00.063601 8800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:00.065630 8800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:00.065722 8800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:00.065867 8800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:00.066087 8800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:00.066119 8800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.066138 8800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:00.066148 8800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.071162 8800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45923
I20260925 19:01:00.071223 8928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45923 every 8 connection(s)
I20260925 19:01:00.071544 8800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:01:00.074374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 8800
I20260925 19:01:00.074502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:01:00.074704 392 external_mini_cluster.cc:1428] Setting key 7d8bba2c8680c32b70384540679ec157
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:01:00.081238 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.072180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45777 (local address 127.0.98.62:43835)
0925 19:01:00.072259 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.072261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.072430 (+ 169us) server_negotiation.cc:407] Connection header received
0925 19:01:00.073260 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.073261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.073270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.073318 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.073831 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.074401 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.075784 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.075960 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.076029 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.078454 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:00.078456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:00.078456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:00.078462 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:00.079768 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:00.080292 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:00.080293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:00.080293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:00.080340 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:00.080615 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:00.080616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:00.080616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:00.080706 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:00.080771 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:00.081097 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 19:01:00.081166 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:01:00.081456 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.072150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:45777)
0925 19:01:00.072321 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.072352 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.073178 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.073372 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.073375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.073601 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.073766 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.073769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.074509 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.074510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.075708 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.075711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.076058 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.076059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.076194 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.076619 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.076629 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.078371 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:00.079852 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:00.079858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:00.079861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:00.080217 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:00.080406 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:00.080406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:00.080407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:00.080552 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:00.080788 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:00.080793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:00.080953 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 19:01:00.081068 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 19:01:00.081235 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260925 19:01:00.081956 8929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43835
I20260925 19:01:00.082044 8929 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:00.082213 8929 heartbeater.cc:511] Master 127.0.98.62:43835 requested a full tablet report, sending...
I20260925 19:01:00.082577 8604 ts_manager.cc:194] Registered new tserver with Master: 71bd901d441d4a7b89684e4f85049a21 (127.0.98.2:45923)
I20260925 19:01:00.082917 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45777
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:01:00.098774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:01:00.176144 8934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:00.176324 8934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:00.176347 8934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:00.177320 8934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:00.177351 8934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:00.177373 8934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:01:00.178756 8934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:00.178915 8934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:00.179060 8934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:00.183657 8942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:00.183749 8934 server_base.cc:1034] running on GCE node
W20260925 19:01:00.183669 8940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:00.183668 8939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:00.184131 8934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:00.184348 8934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:00.185482 8934 hybrid_clock.cc:648] HybridClock initialized: now 1790362860185463 us; error 36 us; skew 500 ppm
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:00.187918 8934 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:01:00.188306 8934 webserver.cc:460] Webserver started at http://127.0.98.3:42845/ using document root <none> and password file <none>
I20260925 19:01:00.188499 8934 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:00.188544 8934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:00.188697 8934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:00.189790 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "25130aaa317748a3b5f04fe09d464788"
format_stamp: "Formatted at 2026-09-25 19:01:00 on dist-test-slave-49bz"
server_key: "dfa04e3ca01758be1252fad552855b40"
server_key_iv: "a58f9a3818340d1d7d288de4d517c7de"
server_key_version: "encryptionkey@0"
I20260925 19:01:00.190219 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "25130aaa317748a3b5f04fe09d464788"
format_stamp: "Formatted at 2026-09-25 19:01:00 on dist-test-slave-49bz"
server_key: "dfa04e3ca01758be1252fad552855b40"
server_key_iv: "a58f9a3818340d1d7d288de4d517c7de"
server_key_version: "encryptionkey@0"
I20260925 19:01:00.191454 8934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:01:00.192122 8949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.192257 8934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:01:00.192330 8934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "25130aaa317748a3b5f04fe09d464788"
format_stamp: "Formatted at 2026-09-25 19:01:00 on dist-test-slave-49bz"
server_key: "dfa04e3ca01758be1252fad552855b40"
server_key_iv: "a58f9a3818340d1d7d288de4d517c7de"
server_key_version: "encryptionkey@0"
I20260925 19:01:00.192375 8934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:00.214202 8934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:00.215881 8934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:00.216027 8934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:00.216223 8934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:00.216490 8934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:00.216521 8934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.216540 8934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:00.216583 8934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.222152 8934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41721
I20260925 19:01:00.222218 9062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41721 every 8 connection(s)
I20260925 19:01:00.222568 8934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:01:00.222841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 8934
I20260925 19:01:00.222939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:01:00.223155 392 external_mini_cluster.cc:1428] Setting key f58a64168a3d72943878d0ff78af716a
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:01:00.231145 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.223242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52175 (local address 127.0.98.62:43835)
0925 19:01:00.223294 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.223296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.223525 (+ 229us) server_negotiation.cc:407] Connection header received
0925 19:01:00.224629 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.224629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.224639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.224683 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.225220 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.225696 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.226757 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.226930 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.226981 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.228923 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:00.228924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:00.228924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:00.228930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:00.230043 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:00.230428 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:00.230430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:00.230430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:00.230478 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:00.230678 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:00.230679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:00.230679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:00.230763 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:00.230807 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:00.231031 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 19:01:00.231086 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:01:00.231350 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.223238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:52175)
0925 19:01:00.223415 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.223458 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.224532 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.224732 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.224736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.224954 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.225147 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.225151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.225769 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.225769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.226694 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.226696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.226995 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.226995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.227094 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.227495 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.227502 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.228833 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:00.230108 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:00.230114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:00.230115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:00.230369 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:00.230547 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:00.230548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:00.230548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:00.230636 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:00.230808 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:00.230812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:00.230910 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 19:01:00.231004 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 19:01:00.231115 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260925 19:01:00.231775 9063 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43835
I20260925 19:01:00.231849 9063 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:00.232035 9063 heartbeater.cc:511] Master 127.0.98.62:43835 requested a full tablet report, sending...
I20260925 19:01:00.232376 8604 ts_manager.cc:194] Registered new tserver with Master: 25130aaa317748a3b5f04fe09d464788 (127.0.98.3:41721)
I20260925 19:01:00.232725 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52175
I20260925 19:01:00.234781 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 19:01:00.297662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 8666
I20260925 19:01:00.301959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 8800
I20260925 19:01:00.306413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 8934
I20260925 19:01:00.310216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 8572
I20260925 19:01:00.315003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43835
--webserver_interface=127.0.98.62
--webserver_port=42265
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:01:00.393939 9072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:00.394162 9072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:00.394200 9072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:00.395351 9072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:01:00.395416 9072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:00.395442 9072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:00.395471 9072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:01:00.395495 9072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:01:00.396903 9072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:00.397158 9072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:00.397359 9072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:00.402091 9080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:00.402221 9077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:00.402271 9072 server_base.cc:1034] running on GCE node
W20260925 19:01:00.402329 9078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:00.402591 9072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:00.402824 9072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:00.403959 9072 hybrid_clock.cc:648] HybridClock initialized: now 1790362860403927 us; error 48 us; skew 500 ppm
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:00.406188 9072 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:01:00.406648 9072 webserver.cc:460] Webserver started at http://127.0.98.62:42265/ using document root <none> and password file <none>
I20260925 19:01:00.406868 9072 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:00.406947 9072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:00.408258 9072 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:01:00.409016 9087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.409214 9072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:01:00.409318 9072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "d94d2154b58e46bfbc0ec938b771a248"
format_stamp: "Formatted at 2026-09-25 19:00:59 on dist-test-slave-49bz"
server_key: "0ae8a81442ab105b13ab92d456bd23f7"
server_key_iv: "d41379e1fb1408e2ddc76fe6b41fdb48"
server_key_version: "encryptionkey@0"
I20260925 19:01:00.409624 9072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:00.444460 9072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:00.446223 9072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:00.446398 9072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:00.450181 9072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43835
I20260925 19:01:00.450213 9139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43835 every 8 connection(s)
I20260925 19:01:00.450755 9072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:01:00.451900 9140 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:01:00.452514 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: Bootstrap starting.
I20260925 19:01:00.459183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 9072
I20260925 19:01:00.459467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38333
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43835
--webserver_port=33717
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:01:00.460503 9140 log.cc:824] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: Log is configured to *not* fsync() on all Append() calls
I20260925 19:01:00.461998 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:01:00.462240 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: Bootstrap complete.
I20260925 19:01:00.464325 9140 raft_consensus.cc:348] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260925 19:01:00.464555 9140 raft_consensus.cc:729] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94d2154b58e46bfbc0ec938b771a248, State: Initialized, Role: FOLLOWER
I20260925 19:01:00.464752 9140 consensus_queue.cc:260] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260925 19:01:00.464823 9140 raft_consensus.cc:388] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:01:00.464867 9140 raft_consensus.cc:482] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:01:00.464910 9140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:01:00.465802 9140 raft_consensus.cc:504] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260925 19:01:00.465917 9140 leader_election.cc:302] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d94d2154b58e46bfbc0ec938b771a248; no voters:
I20260925 19:01:00.466060 9140 leader_election.cc:288] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:01:00.466105 9144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:01:00.466238 9144 raft_consensus.cc:686] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [term 2 LEADER]: Becoming Leader. State: Replica: d94d2154b58e46bfbc0ec938b771a248, State: Running, Role: LEADER
I20260925 19:01:00.466327 9140 sys_catalog.cc:564] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:01:00.466352 9144 consensus_queue.cc:237] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260925 19:01:00.466687 9144 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d94d2154b58e46bfbc0ec938b771a248. Latest consensus state: current_term: 2 leader_uuid: "d94d2154b58e46bfbc0ec938b771a248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } } }
I20260925 19:01:00.466787 9144 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [sys.catalog]: This master's current role is: LEADER
I20260925 19:01:00.466687 9145 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d94d2154b58e46bfbc0ec938b771a248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94d2154b58e46bfbc0ec938b771a248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } } }
I20260925 19:01:00.466957 9145 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248 [sys.catalog]: This master's current role is: LEADER
I20260925 19:01:00.467428 9151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:01:00.467942 9151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:01:00.468098 9151 catalog_manager.cc:1259] Loaded cluster ID: 100efdb8651b4e47a9b9e2b1c59a85cb
I20260925 19:01:00.468142 9151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:01:00.469409 9151 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:01:00.469568 9151 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d94d2154b58e46bfbc0ec938b771a248: Loaded TSK: 0
I20260925 19:01:00.469763 9151 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:01:00.537684 9142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:00.537858 9142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:00.537879 9142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:00.539124 9142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:00.539165 9142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:00.539191 9142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:01:00.540522 9142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33717
--tserver_master_addrs=127.0.98.62:43835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:00.540735 9142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:00.540956 9142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:00.545675 9166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:00.545655 9169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:00.545862 9142 server_base.cc:1034] running on GCE node
W20260925 19:01:00.545825 9167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:00.546144 9142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:00.546425 9142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:00.547618 9142 hybrid_clock.cc:648] HybridClock initialized: now 1790362860547609 us; error 29 us; skew 500 ppm
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:00.549973 9142 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 19:01:00.550505 9142 webserver.cc:460] Webserver started at http://127.0.98.1:33717/ using document root <none> and password file <none>
I20260925 19:01:00.550734 9142 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:00.550787 9142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:00.552042 9142 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:01:00.552631 9176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.552773 9142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:01:00.552851 9142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "6b8305d23fe04c588049517e109829b4"
format_stamp: "Formatted at 2026-09-25 19:00:59 on dist-test-slave-49bz"
server_key: "99d8c1969ffb01aa032e227717e0576d"
server_key_iv: "81c9b8a0fbeda8cbfc1cd2925343dc2e"
server_key_version: "encryptionkey@0"
I20260925 19:01:00.553205 9142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:00.563210 9142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:00.565234 9142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:00.565371 9142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:00.565635 9142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:00.565969 9142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:00.566008 9142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.566030 9142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:00.566042 9142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.572849 9142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38333
I20260925 19:01:00.572917 9289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38333 every 8 connection(s)
I20260925 19:01:00.573197 9142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:01:00.573948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 9142
I20260925 19:01:00.574236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45923
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43835
--webserver_port=45013
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:00.583348 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.573890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:50019)
0925 19:01:00.574468 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.574512 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.574996 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.575396 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.575399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.575901 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.576056 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.576060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.577488 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.577489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.578728 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.578731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.579595 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.579597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.579729 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.580311 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.580322 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.583132 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":84,"threads_started":1}
W20260925 19:01:00.583410 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:00.573927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50019 (local address 127.0.98.62:43835)
0925 19:01:00.574162 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.574170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.574621 (+ 451us) server_negotiation.cc:407] Connection header received
0925 19:01:00.575187 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.575193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.575239 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.575362 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.576187 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.577379 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.578820 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.579498 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.579627 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.583255 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50019: BlockingRecv error: recv got EOF from 127.0.98.1:50019 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":64,"threads_started":1}
W20260925 19:01:00.583676 9290 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43835 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43835: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:00.588886 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.583909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:34337)
0925 19:01:00.583958 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.583969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.584019 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.584197 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.584198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.584354 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.584405 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.584406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.585248 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.585249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.586052 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.586054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.586547 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.586548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.586616 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.587029 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.587034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.588825 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:01:00.588965 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:00.583934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34337 (local address 127.0.98.62:43835)
0925 19:01:00.584082 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.584084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.584090 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:01:00.584109 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.584110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.584120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.584168 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.584485 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.585177 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.586144 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.586476 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.586548 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.588910 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34337: BlockingRecv error: recv got EOF from 127.0.98.1:34337 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:00.593451 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.589326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:34011)
0925 19:01:00.589385 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.589396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.589436 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.589555 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.589556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.589677 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.589703 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.589704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.590576 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.590576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.591016 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.591017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.591469 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.591469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.591501 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.591875 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.591879 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.593374 (+ 1495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 19:01:00.593475 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:00.589380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34011 (local address 127.0.98.62:43835)
0925 19:01:00.589411 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.589413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.589423 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:01:00.589482 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.589482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.589491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.589542 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.589775 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.590506 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.591083 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.591395 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.591468 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.593439 (+ 1971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34011: BlockingRecv error: recv got EOF from 127.0.98.1:34011 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 19:01:00.593721 9290 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:01:00.655495 9294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:00.655704 9294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:00.655725 9294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:00.656764 9294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:00.656800 9294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:00.656823 9294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:01:00.658010 9294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45013
--tserver_master_addrs=127.0.98.62:43835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:00.658164 9294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:00.658325 9294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:00.662869 9302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:00.662891 9300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:00.662901 9299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:00.663139 9294 server_base.cc:1034] running on GCE node
I20260925 19:01:00.663321 9294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:00.663561 9294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:00.664736 9294 hybrid_clock.cc:648] HybridClock initialized: now 1790362860664724 us; error 28 us; skew 500 ppm
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:00.667132 9294 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 19:01:00.667538 9294 webserver.cc:460] Webserver started at http://127.0.98.2:45013/ using document root <none> and password file <none>
I20260925 19:01:00.667805 9294 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:00.667892 9294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:00.669080 9294 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:01:00.669588 9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.669756 9294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:01:00.669835 9294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "71bd901d441d4a7b89684e4f85049a21"
format_stamp: "Formatted at 2026-09-25 19:01:00 on dist-test-slave-49bz"
server_key: "57a19006acaae9015a126f6a4db4eb7d"
server_key_iv: "362caad54cccf2099c4f77cd1e306e6c"
server_key_version: "encryptionkey@0"
I20260925 19:01:00.670122 9294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:00.682829 9294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:00.684556 9294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:00.684712 9294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:00.684916 9294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:00.685199 9294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:00.685242 9294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.685261 9294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:00.685305 9294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.691175 9294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45923
I20260925 19:01:00.691210 9422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45923 every 8 connection(s)
I20260925 19:01:00.691526 9294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 19:01:00.698215 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:00.692172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52165 (local address 127.0.98.62:43835)
0925 19:01:00.692221 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.692223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.692409 (+ 186us) server_negotiation.cc:407] Connection header received
0925 19:01:00.692876 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.692877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.692891 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.692941 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.693809 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.694363 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.695287 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.695475 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.695556 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.698162 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52165: BlockingRecv error: recv got EOF from 127.0.98.2:52165 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:01:00.698293 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.692168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:52165)
0925 19:01:00.692301 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.692331 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.692777 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.693005 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.693008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.693541 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.693707 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.693711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.694445 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.694446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.695194 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.695196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.695564 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.695564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.695672 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.696166 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.696173 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.698102 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
W20260925 19:01:00.698498 9423 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43835 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43835: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260925 19:01:00.699313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 9294
I20260925 19:01:00.699584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41721
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43835
--webserver_port=42845
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:00.702721 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.698695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:46143)
0925 19:01:00.698741 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.698750 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.698787 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.698897 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.698898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.699020 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.699055 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.699056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.699781 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.699782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.700271 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.700272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.700661 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.700662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.700723 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.701135 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.701139 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.702655 (+ 1516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 19:01:00.702766 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:00.698752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46143 (local address 127.0.98.62:43835)
0925 19:01:00.698791 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.698793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.698801 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:01:00.698841 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.698842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.698851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.698896 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.699127 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.699685 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.700374 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.700576 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.700651 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.702714 (+ 2063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46143: BlockingRecv error: recv got EOF from 127.0.98.2:46143 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:00.706929 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.702994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:46161)
0925 19:01:00.703027 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.703041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.703081 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.703236 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.703236 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.703348 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.703369 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.703370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.703955 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.703956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.704418 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.704420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.704784 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.704785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.704834 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.705220 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.705225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.706849 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 19:01:00.706949 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:00.703049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46161 (local address 127.0.98.62:43835)
0925 19:01:00.703103 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.703105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.703113 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:01:00.703131 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.703131 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.703156 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.703231 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.703448 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.703880 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.704509 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.704699 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.704783 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.706913 (+ 2130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46161: BlockingRecv error: recv got EOF from 127.0.98.2:46161 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 19:01:00.707077 9423 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:01:00.775813 9426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:00.776010 9426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:00.776063 9426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:00.777117 9426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:00.777175 9426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:00.777217 9426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:01:00.778573 9426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42845
--tserver_master_addrs=127.0.98.62:43835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:00.778836 9426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:00.779069 9426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:00.783713 9432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:00.783682 9434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:00.783715 9431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:00.783986 9426 server_base.cc:1034] running on GCE node
I20260925 19:01:00.784132 9426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:00.784298 9426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:00.785424 9426 hybrid_clock.cc:648] HybridClock initialized: now 1790362860785417 us; error 27 us; skew 500 ppm
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:00.787680 9426 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 19:01:00.788056 9426 webserver.cc:460] Webserver started at http://127.0.98.3:42845/ using document root <none> and password file <none>
I20260925 19:01:00.788268 9426 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:00.788319 9426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:00.789569 9426 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:01:00.790079 9441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.790238 9426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:01:00.790364 9426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "25130aaa317748a3b5f04fe09d464788"
format_stamp: "Formatted at 2026-09-25 19:01:00 on dist-test-slave-49bz"
server_key: "dfa04e3ca01758be1252fad552855b40"
server_key_iv: "a58f9a3818340d1d7d288de4d517c7de"
server_key_version: "encryptionkey@0"
I20260925 19:01:00.790668 9426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:00.804916 9426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:00.806540 9426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:00.806677 9426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:00.806865 9426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:00.807148 9426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:00.807188 9426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.807224 9426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:00.807250 9426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:00.812765 9426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41721
I20260925 19:01:00.812799 9554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41721 every 8 connection(s)
I20260925 19:01:00.813109 9426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:01:00.814200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 9426
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 19:01:00.820304 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:00.813784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59595 (local address 127.0.98.62:43835)
0925 19:01:00.813827 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.813829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.814006 (+ 177us) server_negotiation.cc:407] Connection header received
0925 19:01:00.814875 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.814876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.814886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.814938 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.815765 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.816208 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.817165 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.817360 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.817439 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.820249 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59595: BlockingRecv error: recv got EOF from 127.0.98.3:59595 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:01:00.820444 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.813757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:59595)
0925 19:01:00.813905 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.813941 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.814796 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.815014 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.815017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.815513 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.815675 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.815679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.816309 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.816310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.817042 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.817044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.817450 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.817451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.817563 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.817995 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.818002 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.820190 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
W20260925 19:01:00.820637 9555 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43835 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43835: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260925 19:01:00.821494 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.814675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.0.98.62:43835)
0925 19:01:00.814791 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.814796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.814803 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:01:00.814840 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.814841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.814850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.814921 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.815516 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.815992 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.816755 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.816942 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.817012 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.818379 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:00.818383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:00.818384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:00.818413 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:00.820012 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:00.820396 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:00.820399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:00.820399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:00.820516 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:00.820724 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:00.820725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:00.820725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:00.821000 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:00.821158 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:00.821265 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 19:01:00.821404 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":80,"threads_started":1}
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:00.824852 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.820844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:47151)
0925 19:01:00.820889 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.820899 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.820938 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.821051 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.821052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.821188 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.821224 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.821225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.822124 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.822125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.822713 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.822715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.823008 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.823009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.823055 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.823446 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.823451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.824813 (+ 1362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:01:00.824967 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:00.820903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47151 (local address 127.0.98.62:43835)
0925 19:01:00.820959 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.820961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.820981 (+ 20us) server_negotiation.cc:407] Connection header received
0925 19:01:00.820998 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.820998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.821006 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.821050 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.821330 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.821993 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.822778 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.822948 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.822997 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.824900 (+ 1903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47151: BlockingRecv error: recv got EOF from 127.0.98.3:47151 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:00 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:00.828760 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:00.825247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:41265)
0925 19:01:00.825308 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:00.825317 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:00.825365 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:00.825455 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:00.825456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:00.825566 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:00.825587 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.825588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.826123 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.826124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.826641 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:00.826642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.826923 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.826924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:00.826955 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.827246 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:00.827250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:00.828721 (+ 1471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 19:01:00.828864 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:00.825299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41265 (local address 127.0.98.62:43835)
0925 19:01:00.825330 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:00.825332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:00.825355 (+ 23us) server_negotiation.cc:407] Connection header received
0925 19:01:00.825402 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:00.825402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:00.825408 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:00.825444 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:00.825648 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.826062 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.826701 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:00.826864 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:00.826912 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:00.828785 (+ 1873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41265: BlockingRecv error: recv got EOF from 127.0.98.3:41265 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 19:01:00.828877 9555 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 19:01:01 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:01 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:01.598304 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:01.594103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:35157)
0925 19:01:01.594224 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:01.594236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:01.594310 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:01.594443 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:01.594444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:01.594593 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:01.594639 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:01.594640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.595237 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.595238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:01.595727 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:01.595728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.596087 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.596088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:01.596125 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:01.596607 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:01.596611 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:01.598189 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
W20260925 19:01:01.598424 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:01.594162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35157 (local address 127.0.98.62:43835)
0925 19:01:01.594303 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:01.594305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:01.594313 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:01:01.594371 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:01.594371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:01.594380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:01.594433 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:01.594696 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.595179 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.595795 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.595997 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.596072 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:01.598331 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35157: BlockingRecv error: recv got EOF from 127.0.98.1:35157 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
Sep 25 19:01:01 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:01 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:01.711972 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:01.707524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:35659)
0925 19:01:01.707645 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:01.707655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:01.707715 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:01.707827 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:01.707827 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:01.707967 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:01.708012 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:01.708013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.708598 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.708599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:01.709099 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:01.709101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.709416 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.709418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:01.709463 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:01.709884 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:01.709888 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:01.711881 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
W20260925 19:01:01.712003 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:01.707588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35659 (local address 127.0.98.62:43835)
0925 19:01:01.707653 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:01.707655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:01.707716 (+ 61us) server_negotiation.cc:407] Connection header received
0925 19:01:01.707753 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:01.707754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:01.707763 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:01.707810 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:01.708075 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.708526 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.709167 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.709333 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.709384 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:01.711947 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35659: BlockingRecv error: recv got EOF from 127.0.98.2:35659 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 19:01:01 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:01 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:01.833705 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:01.829222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:34765)
0925 19:01:01.829398 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:01.829410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:01.829500 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:01.829638 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:01.829639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:01.829815 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:01.829859 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:01.829861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.830560 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.830561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:01.831117 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:01.831119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.831489 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.831490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:01.831546 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:01.831960 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:01.831964 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:01.833582 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
W20260925 19:01:01.833739 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:01.829292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34765 (local address 127.0.98.62:43835)
0925 19:01:01.829381 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:01.829383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:01.829482 (+ 99us) server_negotiation.cc:407] Connection header received
0925 19:01:01.829555 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:01.829555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:01.829571 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:01.829615 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:01.829936 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.830454 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.831211 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:01.831409 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:01.831489 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:01.833675 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34765: BlockingRecv error: recv got EOF from 127.0.98.3:34765 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 19:01:02 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:02 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:02.603137 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:02.598677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:46573)
0925 19:01:02.598822 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:02.598831 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:02.598887 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:02.599065 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:02.599066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:02.599209 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:02.599273 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:02.599275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.599874 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.599875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:02.600426 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:02.600428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.600724 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.600725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:02.600762 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:02.601332 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:02.601336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:02.603078 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
W20260925 19:01:02.603240 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:02.598745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46573 (local address 127.0.98.62:43835)
0925 19:01:02.598832 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:02.598836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:02.598872 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:01:02.598936 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:02.598937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:02.598946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:02.599003 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:02.599339 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.599807 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.600497 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.600666 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.600713 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:02.603165 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46573: BlockingRecv error: recv got EOF from 127.0.98.1:46573 (error 108)
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":45,"threads_started":1}
Sep 25 19:01:02 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:02 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:02.716733 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:02.712355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:37405)
0925 19:01:02.712484 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:02.712500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:02.712549 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:02.712695 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:02.712696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:02.712839 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:02.712886 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:02.712887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.713466 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.713467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:02.714010 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:02.714011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.714462 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.714463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:02.714544 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:02.715068 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:02.715072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:02.716669 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
W20260925 19:01:02.716861 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:02.712430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37405 (local address 127.0.98.62:43835)
0925 19:01:02.712492 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:02.712494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:02.712545 (+ 51us) server_negotiation.cc:407] Connection header received
0925 19:01:02.712613 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:02.712613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:02.712623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:02.712680 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:02.712945 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.713387 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.714082 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.714383 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.714451 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:02.716780 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37405: BlockingRecv error: recv got EOF from 127.0.98.2:37405 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 19:01:02 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:02 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:02.838708 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:02.834179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:47261)
0925 19:01:02.834324 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:02.834336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:02.834399 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:02.834568 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:02.834568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:02.834744 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:02.834789 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:02.834791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.835520 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.835521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:02.836051 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:02.836053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.836421 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.836421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:02.836475 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:02.836884 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:02.836888 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:02.838644 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
W20260925 19:01:02.838801 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:02.834341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47261 (local address 127.0.98.62:43835)
0925 19:01:02.834391 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:02.834393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:02.834401 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:01:02.834465 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:02.834466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:02.834478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:02.834549 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:02.834872 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.835444 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.836138 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:02.836325 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:02.836409 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:02.838742 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47261: BlockingRecv error: recv got EOF from 127.0.98.3:47261 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 19:01:03 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:03 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:03.607838 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:03.603573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:47219)
0925 19:01:03.603702 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:03.603712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:03.603766 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:03.603921 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:03.603922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:03.604087 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:03.604133 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:03.604134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.604740 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.604740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:03.605277 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:03.605279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.605580 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.605581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:03.605619 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:03.606022 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:03.606026 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:03.607782 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
W20260925 19:01:03.607957 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:03.603650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47219 (local address 127.0.98.62:43835)
0925 19:01:03.603814 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:03.603816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:03.603823 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:01:03.603842 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:03.603843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:03.603852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:03.603906 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:03.604198 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.604680 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.605346 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.605521 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.605569 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:03.607875 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47219: BlockingRecv error: recv got EOF from 127.0.98.1:47219 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
Sep 25 19:01:03 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:03 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:03.721441 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:03.717147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:50063)
0925 19:01:03.717291 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:03.717302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:03.717363 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:03.717506 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:03.717507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:03.717680 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:03.717726 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:03.717728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.718402 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.718404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:03.718924 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:03.718926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.719256 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.719257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:03.719310 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:03.719763 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:03.719767 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:03.721389 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
W20260925 19:01:03.721570 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:03.717215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50063 (local address 127.0.98.62:43835)
0925 19:01:03.717272 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:03.717275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:03.717345 (+ 70us) server_negotiation.cc:407] Connection header received
0925 19:01:03.717417 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:03.717418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:03.717430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:03.717499 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:03.717803 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.718329 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.719002 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.719180 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.719260 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:03.721492 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50063: BlockingRecv error: recv got EOF from 127.0.98.2:50063 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 19:01:03 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:03 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:03.843290 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:03.839218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:33529)
0925 19:01:03.839359 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:03.839369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:03.839428 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:03.839533 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:03.839533 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:03.839685 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:03.839731 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:03.839732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.840333 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.840334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:03.840836 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:03.840837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.841142 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.841143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:03.841180 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:03.841594 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:03.841598 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:03.843236 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
W20260925 19:01:03.843412 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:03.839277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33529 (local address 127.0.98.62:43835)
0925 19:01:03.839327 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:03.839329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:03.839432 (+ 103us) server_negotiation.cc:407] Connection header received
0925 19:01:03.839477 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:03.839478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:03.839490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:03.839528 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:03.839789 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.840268 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.840906 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:03.841084 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:03.841135 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:03.843317 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33529: BlockingRecv error: recv got EOF from 127.0.98.3:33529 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 19:01:04 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:04 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:04.612680 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:04.608382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:60463)
0925 19:01:04.608540 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:04.608551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:04.608617 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:04.608735 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:04.608735 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:04.608903 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:04.608948 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:04.608949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.609527 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.609528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:04.609994 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:04.609995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.610378 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.610380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:04.610428 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:04.610847 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:04.610852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:04.612586 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
W20260925 19:01:04.612717 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:04.608428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60463 (local address 127.0.98.62:43835)
0925 19:01:04.608562 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:04.608564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:04.608604 (+ 40us) server_negotiation.cc:407] Connection header received
0925 19:01:04.608657 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:04.608658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:04.608667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:04.608719 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:04.609006 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.609462 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.610056 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.610304 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.610354 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:04.612654 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60463: BlockingRecv error: recv got EOF from 127.0.98.1:60463 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
Sep 25 19:01:04 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:04 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:04.726517 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:04.721837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:41469)
0925 19:01:04.722007 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:04.722019 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:04.722103 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:04.722293 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:04.722294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:04.722456 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:04.722500 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:04.722502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.723182 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.723183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:04.723689 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:04.723691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.724069 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.724070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:04.724136 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:04.724638 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:04.724643 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:04.726426 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
W20260925 19:01:04.726567 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:04.721909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41469 (local address 127.0.98.62:43835)
0925 19:01:04.722011 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:04.722013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:04.722086 (+ 73us) server_negotiation.cc:407] Connection header received
0925 19:01:04.722190 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:04.722191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:04.722203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:04.722245 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:04.722584 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.723093 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.723781 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.723999 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.724072 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:04.726506 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41469: BlockingRecv error: recv got EOF from 127.0.98.2:41469 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 19:01:04 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:04 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:04.847972 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:04.843747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:50045)
0925 19:01:04.843876 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:04.843887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:04.843971 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:04.844092 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:04.844093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:04.844249 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:04.844297 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:04.844298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.844896 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.844897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:04.845449 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:04.845450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.845743 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.845743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:04.845779 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:04.846286 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:04.846289 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:04.847919 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
W20260925 19:01:04.848075 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:04.843814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50045 (local address 127.0.98.62:43835)
0925 19:01:04.843875 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:04.843877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:04.843970 (+ 93us) server_negotiation.cc:407] Connection header received
0925 19:01:04.844011 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:04.844012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:04.844023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:04.844072 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:04.844354 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.844826 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.845518 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:04.845683 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:04.845730 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:04.848005 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50045: BlockingRecv error: recv got EOF from 127.0.98.3:50045 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 19:01:05 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:05 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:05.622684 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:05.613154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:39537)
0925 19:01:05.613281 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:05.613290 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:05.613337 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:05.613451 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:05.613452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:05.613596 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:05.613642 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:05.613653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.614331 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.614331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:05.614860 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:05.614862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.615189 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.615189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:05.615226 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:05.620852 (+ 5626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:05.620865 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:05.622618 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
W20260925 19:01:05.622776 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:05.613196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39537 (local address 127.0.98.62:43835)
0925 19:01:05.613335 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:05.613338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:05.613346 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:01:05.613369 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:05.613370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:05.613380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:05.613438 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:05.613714 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.614218 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.614925 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.615112 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.615175 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:05.622707 (+ 7532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39537: BlockingRecv error: recv got EOF from 127.0.98.1:39537 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
Sep 25 19:01:05 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:05 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:05.730945 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:05.726931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:57131)
0925 19:01:05.727081 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:05.727098 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:05.727161 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:05.727262 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:05.727263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:05.727408 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:05.727452 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:05.727453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.728012 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.728013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:05.728491 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:05.728493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.728784 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.728784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:05.728820 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:05.729272 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:05.729276 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:05.730891 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
W20260925 19:01:05.731024 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:05.727062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57131 (local address 127.0.98.62:43835)
0925 19:01:05.727104 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:05.727105 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:05.727146 (+ 41us) server_negotiation.cc:407] Connection header received
0925 19:01:05.727212 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:05.727212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:05.727222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:05.727257 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:05.727506 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.727951 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.728557 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.728725 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.728771 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:05.730980 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57131: BlockingRecv error: recv got EOF from 127.0.98.2:57131 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 19:01:05 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:05 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:05.852468 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:05.848537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:56823)
0925 19:01:05.848662 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:05.848671 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:05.848727 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:05.848839 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:05.848840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:05.848985 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:05.849028 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:05.849030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.849575 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.849576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:05.850050 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:05.850051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.850358 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.850359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:05.850397 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:05.850801 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:05.850805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:05.852403 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
W20260925 19:01:05.852574 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:05.848589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56823 (local address 127.0.98.62:43835)
0925 19:01:05.848632 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:05.848634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:05.848727 (+ 93us) server_negotiation.cc:407] Connection header received
0925 19:01:05.848765 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:05.848766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:05.848776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:05.848820 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:05.849080 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.849516 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.850108 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:05.850306 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:05.850350 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:05.852486 (+ 2136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56823: BlockingRecv error: recv got EOF from 127.0.98.3:56823 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 19:01:06 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:06 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:06.627633 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:06.623166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:54469)
0925 19:01:06.623314 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:06.623323 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:06.623365 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:06.623509 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:06.623509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:06.623648 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:06.623691 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:06.623692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.624293 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.624294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:06.624825 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:06.624827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.625150 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.625151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:06.625202 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:06.625706 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:06.625711 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:06.627579 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
W20260925 19:01:06.627770 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:06.623262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54469 (local address 127.0.98.62:43835)
0925 19:01:06.623383 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:06.623386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:06.623411 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:01:06.623430 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:06.623431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:06.623439 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:06.623496 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:06.623750 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.624230 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.624897 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.625075 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.625125 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:06.627676 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54469: BlockingRecv error: recv got EOF from 127.0.98.1:54469 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
Sep 25 19:01:06 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:06 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:06.740617 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:06.731425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:38117)
0925 19:01:06.731586 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:06.731598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:06.731682 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:06.731831 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:06.731832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:06.732001 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:06.732044 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:06.732045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.732719 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.732720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:06.733463 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:06.733465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.733868 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.733869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:06.733912 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:06.738731 (+ 4819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:06.738736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:06.740551 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
W20260925 19:01:06.740746 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:06.731509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38117 (local address 127.0.98.62:43835)
0925 19:01:06.731583 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:06.731585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:06.731665 (+ 80us) server_negotiation.cc:407] Connection header received
0925 19:01:06.731750 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:06.731751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:06.731763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:06.731819 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:06.732119 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.732639 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.733552 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.733808 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.733867 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:06.740641 (+ 6774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38117: BlockingRecv error: recv got EOF from 127.0.98.2:38117 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 19:01:06 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:06 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:06.857704 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:06.852873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:57723)
0925 19:01:06.852987 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:06.852996 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:06.853073 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:06.853209 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:06.853209 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:06.853379 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:06.853423 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:06.853424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.854070 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.854072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:06.854720 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:06.854721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.855095 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.855096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:06.855152 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:06.855730 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:06.855734 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:06.857648 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
W20260925 19:01:06.857839 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:06.852908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57723 (local address 127.0.98.62:43835)
0925 19:01:06.852954 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:06.852957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:06.853054 (+ 97us) server_negotiation.cc:407] Connection header received
0925 19:01:06.853131 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:06.853132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:06.853145 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:06.853187 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:06.853505 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.853988 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.854805 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:06.855009 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:06.855093 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:06.857759 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57723: BlockingRecv error: recv got EOF from 127.0.98.3:57723 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 19:01:07 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:07 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:07.632390 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:07.628099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:48079)
0925 19:01:07.628241 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:07.628255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:07.628317 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:07.628435 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:07.628436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:07.628582 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:07.628627 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:07.628629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.629297 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.629298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:07.629805 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:07.629807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.630115 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.630116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:07.630170 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:07.630712 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:07.630717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:07.632326 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
W20260925 19:01:07.632499 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:07.628154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48079 (local address 127.0.98.62:43835)
0925 19:01:07.628271 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:07.628273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:07.628302 (+ 29us) server_negotiation.cc:407] Connection header received
0925 19:01:07.628365 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:07.628366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:07.628375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:07.628421 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:07.628728 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.629222 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.629876 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.630052 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.630099 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:07.632426 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48079: BlockingRecv error: recv got EOF from 127.0.98.1:48079 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
Sep 25 19:01:07 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:07 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:07.745939 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:07.741012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:37531)
0925 19:01:07.741183 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:07.741195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:07.741264 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:07.741414 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:07.741414 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:07.741595 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:07.741650 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:07.741652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.742410 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.742411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:07.743380 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:07.743383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.743734 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.743735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:07.743772 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:07.744212 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:07.744216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:07.745885 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
W20260925 19:01:07.746064 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:07.741088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37531 (local address 127.0.98.62:43835)
0925 19:01:07.741170 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:07.741173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:07.741245 (+ 72us) server_negotiation.cc:407] Connection header received
0925 19:01:07.741321 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:07.741322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:07.741334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:07.741408 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:07.741732 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.742245 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.743456 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.743665 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.743723 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:07.745990 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37531: BlockingRecv error: recv got EOF from 127.0.98.2:37531 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 19:01:07 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:07 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:07.862478 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:07.858203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:54407)
0925 19:01:07.858365 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:07.858377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:07.858436 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:07.858585 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:07.858585 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:07.858741 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:07.858788 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:07.858789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.859384 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.859386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:07.859874 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:07.859876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.860219 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.860220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:07.860261 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:07.860687 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:07.860692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:07.862425 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
W20260925 19:01:07.862569 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:07.858333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54407 (local address 127.0.98.62:43835)
0925 19:01:07.858385 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:07.858387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:07.858419 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:01:07.858499 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:07.858500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:07.858514 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:07.858565 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:07.858851 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.859307 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.859952 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:07.860112 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:07.860175 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:07.862506 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54407: BlockingRecv error: recv got EOF from 127.0.98.3:54407 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 19:01:08 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:08 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:08.637082 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:08.632833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:52395)
0925 19:01:08.632978 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:08.632988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:08.633030 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:08.633176 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:08.633177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:08.633325 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:08.633370 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:08.633371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.633956 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.633957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:08.634528 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:08.634529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.634841 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.634842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:08.634879 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:08.635294 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:08.635298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:08.636981 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260925 19:01:08.637133 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:08.632944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52395 (local address 127.0.98.62:43835)
0925 19:01:08.633051 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:08.633052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:08.633058 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:01:08.633077 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:08.633077 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:08.633087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:08.633140 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:08.633430 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.633889 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.634594 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.634776 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.634827 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:08.637066 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52395: BlockingRecv error: recv got EOF from 127.0.98.1:52395 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
Sep 25 19:01:08 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:08 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:08.750747 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:08.746455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:57151)
0925 19:01:08.746593 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:08.746604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:08.746675 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:08.746793 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:08.746794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:08.746968 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:08.747009 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:08.747011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.747811 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.747813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:08.748307 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:08.748309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.748653 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.748654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:08.748708 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:08.749118 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:08.749122 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:08.750690 (+ 1568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
W20260925 19:01:08.750917 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:08.746464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57151 (local address 127.0.98.62:43835)
0925 19:01:08.746526 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:08.746528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:08.746649 (+ 121us) server_negotiation.cc:407] Connection header received
0925 19:01:08.746730 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:08.746731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:08.746743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:08.746791 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:08.747087 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.747727 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.748388 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.748579 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.748648 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:08.750850 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57151: BlockingRecv error: recv got EOF from 127.0.98.2:57151 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 19:01:08 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:08 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:08.867522 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:08.862979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:37843)
0925 19:01:08.863114 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:08.863126 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:08.863198 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:08.863363 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:08.863364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:08.863565 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:08.863609 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:08.863610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.864272 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.864273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:08.864818 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:08.864820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.865172 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.865173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:08.865229 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:08.865704 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:08.865707 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:08.867468 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
W20260925 19:01:08.867609 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:08.863027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37843 (local address 127.0.98.62:43835)
0925 19:01:08.863089 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:08.863091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:08.863194 (+ 103us) server_negotiation.cc:407] Connection header received
0925 19:01:08.863274 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:08.863274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:08.863286 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:08.863355 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:08.863692 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.864183 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.864912 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:08.865099 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:08.865173 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:08.867547 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37843: BlockingRecv error: recv got EOF from 127.0.98.3:37843 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 19:01:09 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:09 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:09.641906 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:09.637560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:33213)
0925 19:01:09.637722 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:09.637732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:09.637782 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:09.637932 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:09.637933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:09.638085 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:09.638130 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:09.638131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.638777 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.638778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:09.639309 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:09.639311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.639607 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.639608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:09.639645 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:09.640115 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:09.640119 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:09.641851 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
W20260925 19:01:09.642051 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:09.637659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33213 (local address 127.0.98.62:43835)
0925 19:01:09.637756 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:09.637758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:09.637775 (+ 17us) server_negotiation.cc:407] Connection header received
0925 19:01:09.637843 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:09.637843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:09.637854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:09.637912 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:09.638197 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.638713 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.639376 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.639549 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.639595 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:09.641966 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33213: BlockingRecv error: recv got EOF from 127.0.98.1:33213 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":49,"threads_started":1}
Sep 25 19:01:09 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:09 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:09.755643 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:09.751139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:40101)
0925 19:01:09.751328 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:09.751338 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:09.751406 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:09.751553 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:09.751554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:09.751730 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:09.751774 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:09.751775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.752404 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.752404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:09.752894 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:09.752895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.753255 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.753256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:09.753310 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:09.753747 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:09.753751 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:09.755544 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
W20260925 19:01:09.755682 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:09.751220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40101 (local address 127.0.98.62:43835)
0925 19:01:09.751318 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:09.751320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:09.751387 (+ 67us) server_negotiation.cc:407] Connection header received
0925 19:01:09.751457 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:09.751458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:09.751470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:09.751536 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:09.751851 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.752316 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.752971 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.753155 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.753254 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:09.755628 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40101: BlockingRecv error: recv got EOF from 127.0.98.2:40101 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 19:01:09 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:09 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:09.872304 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:09.868004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:60423)
0925 19:01:09.868154 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:09.868165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:09.868224 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:09.868346 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:09.868346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:09.868492 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:09.868538 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:09.868540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.869127 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.869128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:09.869627 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:09.869628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.869919 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.869920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:09.869955 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:09.870470 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:09.870475 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:09.872215 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
W20260925 19:01:09.872340 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:09.868071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60423 (local address 127.0.98.62:43835)
0925 19:01:09.868154 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:09.868157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:09.868209 (+ 52us) server_negotiation.cc:407] Connection header received
0925 19:01:09.868269 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:09.868269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:09.868281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:09.868328 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:09.868595 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.869066 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.869697 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:09.869863 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:09.869910 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:09.872284 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60423: BlockingRecv error: recv got EOF from 127.0.98.3:60423 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 19:01:10 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:10 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:10.649961 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:10.642384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:51749)
0925 19:01:10.642531 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:10.642544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:10.642603 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:10.642710 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:10.642711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:10.642857 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:10.642919 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:10.642920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.643523 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.643524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:10.644116 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:10.644118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.644495 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.644496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:10.644553 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:10.648065 (+ 3512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:10.648069 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:10.649866 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
W20260925 19:01:10.650014 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:10.642441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51749 (local address 127.0.98.62:43835)
0925 19:01:10.642583 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:10.642584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:10.642592 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:01:10.642643 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:10.642644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:10.642654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:10.642706 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:10.642975 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.643451 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.644218 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.644418 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.644496 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:10.649945 (+ 5449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51749: BlockingRecv error: recv got EOF from 127.0.98.1:51749 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
Sep 25 19:01:10 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:10 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:10.760807 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:10.756212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:42513)
0925 19:01:10.756470 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:10.756481 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:10.756568 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:10.756735 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:10.756736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:10.756902 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:10.756963 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:10.756965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.757605 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.757606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:10.758114 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:10.758116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.758541 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.758542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:10.758602 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:10.759032 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:10.759036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:10.760706 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":174,"threads_started":1}
W20260925 19:01:10.760839 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:10.756387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42513 (local address 127.0.98.62:43835)
0925 19:01:10.756464 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:10.756466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:10.756551 (+ 85us) server_negotiation.cc:407] Connection header received
0925 19:01:10.756633 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:10.756633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:10.756646 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:10.756729 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:10.757041 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.757524 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.758206 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.758464 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.758541 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:10.760779 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42513: BlockingRecv error: recv got EOF from 127.0.98.2:42513 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 19:01:10 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:10 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:10.877125 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:10.872743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:43641)
0925 19:01:10.872881 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:10.872895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:10.872947 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:10.873076 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:10.873077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:10.873242 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:10.873289 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:10.873290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.873909 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.873910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:10.874562 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:10.874564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.874922 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.874922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:10.874964 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:10.875384 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:10.875388 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:10.877066 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
W20260925 19:01:10.877231 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:10.872840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43641 (local address 127.0.98.62:43835)
0925 19:01:10.872888 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:10.872890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:10.872933 (+ 43us) server_negotiation.cc:407] Connection header received
0925 19:01:10.872991 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:10.872992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:10.873005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:10.873059 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:10.873379 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.873846 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.874661 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:10.874855 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:10.874916 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:10.877170 (+ 2254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43641: BlockingRecv error: recv got EOF from 127.0.98.3:43641 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 19:01:11 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:11 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:11.654529 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:11.650444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:37491)
0925 19:01:11.650563 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:11.650574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:11.650625 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:11.650757 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:11.650758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:11.650901 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:11.650946 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:11.650948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.651518 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.651518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:11.652020 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:11.652022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.652322 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.652322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:11.652361 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:11.652779 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:11.652783 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:11.654476 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
W20260925 19:01:11.654752 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:11.650521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37491 (local address 127.0.98.62:43835)
0925 19:01:11.650634 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:11.650636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:11.650644 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:01:11.650672 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:11.650672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:11.650684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:11.650742 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:11.651008 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.651453 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.652088 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.652261 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.652309 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:11.654669 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37491: BlockingRecv error: recv got EOF from 127.0.98.1:37491 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
Sep 25 19:01:11 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:11 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:11.765887 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:11.761252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:56577)
0925 19:01:11.761392 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:11.761437 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:11.761498 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:11.761629 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:11.761630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:11.761787 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:11.761834 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:11.761836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.762446 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.762446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:11.762940 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:11.762941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.763512 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.763513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:11.763575 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:11.764091 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:11.764095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:11.765810 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
W20260925 19:01:11.766004 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:11.761353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56577 (local address 127.0.98.62:43835)
0925 19:01:11.761429 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:11.761432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:11.761497 (+ 65us) server_negotiation.cc:407] Connection header received
0925 19:01:11.761558 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:11.761559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:11.761570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:11.761611 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:11.761893 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.762383 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.763093 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.763450 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.763511 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:11.765931 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56577: BlockingRecv error: recv got EOF from 127.0.98.2:56577 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 19:01:11 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:11 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:11.882105 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:11.877534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:60489)
0925 19:01:11.877688 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:11.877699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:11.877783 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:11.877928 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:11.877929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:11.878100 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:11.878145 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:11.878146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.878821 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.878822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:11.879341 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:11.879342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.879712 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.879713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:11.879772 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:11.880294 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:11.880299 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:11.882033 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
W20260925 19:01:11.882227 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:11.877604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60489 (local address 127.0.98.62:43835)
0925 19:01:11.877669 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:11.877671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:11.877763 (+ 92us) server_negotiation.cc:407] Connection header received
0925 19:01:11.877841 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:11.877842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:11.877861 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:11.877922 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:11.878223 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.878712 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.879434 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:11.879615 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:11.879713 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:11.882132 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60489: BlockingRecv error: recv got EOF from 127.0.98.3:60489 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 19:01:12 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:12 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:12.659314 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:12.655012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:43471)
0925 19:01:12.655193 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:12.655209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:12.655268 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:12.655395 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:12.655395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:12.655551 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:12.655598 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:12.655600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.656250 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.656251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:12.656753 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:12.656754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.657061 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.657062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:12.657098 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:12.657584 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:12.657589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:12.659256 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
W20260925 19:01:12.659452 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:12.655088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43471 (local address 127.0.98.62:43835)
0925 19:01:12.655190 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:12.655192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:12.655253 (+ 61us) server_negotiation.cc:407] Connection header received
0925 19:01:12.655312 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:12.655313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:12.655324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:12.655379 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:12.655665 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.656163 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.656820 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.657000 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.657050 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:12.659362 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43471: BlockingRecv error: recv got EOF from 127.0.98.1:43471 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":45,"threads_started":1}
Sep 25 19:01:12 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:12 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:12.770925 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:12.766356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:41169)
0925 19:01:12.766509 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:12.766520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:12.766594 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:12.766732 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:12.766733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:12.766902 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:12.766979 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:12.766982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.767617 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.767619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:12.768179 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:12.768180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.768558 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.768560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:12.768616 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:12.769033 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:12.769038 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:12.770825 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
W20260925 19:01:12.770951 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:12.766446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41169 (local address 127.0.98.62:43835)
0925 19:01:12.766508 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:12.766510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:12.766567 (+ 57us) server_negotiation.cc:407] Connection header received
0925 19:01:12.766648 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:12.766649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:12.766661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:12.766711 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:12.767053 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.767542 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.768273 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.768474 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.768540 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:12.770893 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41169: BlockingRecv error: recv got EOF from 127.0.98.2:41169 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 19:01:12 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:12 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:12.886775 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:12.882547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:35797)
0925 19:01:12.882664 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:12.882674 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:12.882737 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:12.882852 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:12.882852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:12.882998 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:12.883042 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:12.883043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.883627 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.883628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:12.884122 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:12.884123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.884435 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.884435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:12.884472 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:12.884887 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:12.884892 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:12.886687 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
W20260925 19:01:12.886821 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:12.882589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35797 (local address 127.0.98.62:43835)
0925 19:01:12.882649 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:12.882651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:12.882736 (+ 85us) server_negotiation.cc:407] Connection header received
0925 19:01:12.882795 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:12.882796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:12.882808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:12.882847 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:12.883094 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.883562 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.884193 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:12.884372 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:12.884425 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:12.886756 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35797: BlockingRecv error: recv got EOF from 127.0.98.3:35797 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 19:01:13 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:13 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:13.664042 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:13.659775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:33485)
0925 19:01:13.659922 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:13.659935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:13.659996 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:13.660126 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:13.660126 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:13.660288 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:13.660337 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:13.660339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.660928 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.660929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:13.661437 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:13.661439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.661748 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.661749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:13.661786 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:13.662325 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:13.662329 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:13.663980 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
W20260925 19:01:13.664103 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:13.659829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33485 (local address 127.0.98.62:43835)
0925 19:01:13.659986 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:13.659988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:13.659997 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:01:13.660049 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:13.660049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:13.660059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:13.660105 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:13.660402 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.660869 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.661519 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.661690 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.661735 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:13.664054 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33485: BlockingRecv error: recv got EOF from 127.0.98.1:33485 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":60,"threads_started":1}
Sep 25 19:01:13 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:13 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:13.775579 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:13.771364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:44657)
0925 19:01:13.771475 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:13.771485 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:13.771530 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:13.771663 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:13.771663 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:13.771851 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:13.771897 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:13.771899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.772470 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.772471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:13.773009 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:13.773011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.773308 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.773308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:13.773344 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:13.773827 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:13.773831 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:13.775510 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
W20260925 19:01:13.775641 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:13.771447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44657 (local address 127.0.98.62:43835)
0925 19:01:13.771505 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:13.771511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:13.771520 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:01:13.771585 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:13.771586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:13.771596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:13.771644 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:13.771951 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.772391 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.773073 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.773246 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.773297 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:13.775597 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44657: BlockingRecv error: recv got EOF from 127.0.98.2:44657 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 19:01:13 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:13 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:13.891541 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:13.887308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:60225)
0925 19:01:13.887493 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:13.887522 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:13.887590 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:13.887745 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:13.887745 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:13.887916 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:13.887958 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:13.887959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.888579 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.888580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:13.889061 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:13.889063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.889400 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.889400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:13.889455 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:13.889871 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:13.889874 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:13.891478 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":101,"threads_started":1}
W20260925 19:01:13.891602 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:13.887377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60225 (local address 127.0.98.62:43835)
0925 19:01:13.887454 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:13.887456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:13.887580 (+ 124us) server_negotiation.cc:407] Connection header received
0925 19:01:13.887651 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:13.887651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:13.887670 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:13.887725 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:13.888030 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.888493 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.889150 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:13.889324 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:13.889400 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:13.891559 (+ 2159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60225: BlockingRecv error: recv got EOF from 127.0.98.3:60225 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 19:01:14 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:14 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:14.669162 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:14.664526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:46327)
0925 19:01:14.664712 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:14.664725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:14.664800 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:14.664953 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:14.664953 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:14.665110 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:14.665156 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:14.665157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.665774 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.665774 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:14.666387 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:14.666390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.666800 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.666801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:14.666859 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:14.667311 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:14.667317 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:14.669072 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
W20260925 19:01:14.669230 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:14.664589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46327 (local address 127.0.98.62:43835)
0925 19:01:14.664758 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:14.664760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:14.664778 (+ 18us) server_negotiation.cc:407] Connection header received
0925 19:01:14.664848 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:14.664848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:14.664859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:14.664913 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:14.665218 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.665701 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.666467 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.666703 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.666786 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:14.669149 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46327: BlockingRecv error: recv got EOF from 127.0.98.1:46327 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
Sep 25 19:01:14 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:14 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:14.780045 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:14.776047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:39525)
0925 19:01:14.776192 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:14.776203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:14.776255 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:14.776384 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:14.776385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:14.776532 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:14.776579 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:14.776580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.777167 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.777168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:14.777663 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:14.777664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.777977 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.777978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:14.778013 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:14.778435 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:14.778439 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:14.779978 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
W20260925 19:01:14.780189 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:14.776104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39525 (local address 127.0.98.62:43835)
0925 19:01:14.776182 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:14.776185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:14.776263 (+ 78us) server_negotiation.cc:407] Connection header received
0925 19:01:14.776303 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:14.776304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:14.776314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:14.776366 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:14.776638 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.777106 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.777728 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.777920 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.777967 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:14.780107 (+ 2140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39525: BlockingRecv error: recv got EOF from 127.0.98.2:39525 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 19:01:14 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:14 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:14.896257 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:14.891961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:37211)
0925 19:01:14.892082 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:14.892092 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:14.892142 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:14.892283 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:14.892284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:14.892435 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:14.892480 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:14.892482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.893146 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.893147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:14.893676 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:14.893678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.894006 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.894007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:14.894055 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:14.894537 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:14.894541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:14.896201 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
W20260925 19:01:14.896401 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:14.892032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37211 (local address 127.0.98.62:43835)
0925 19:01:14.892083 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:14.892085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:14.892141 (+ 56us) server_negotiation.cc:407] Connection header received
0925 19:01:14.892184 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:14.892184 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:14.892195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:14.892263 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:14.892537 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.893071 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.893743 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:14.893921 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:14.893973 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:14.896316 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37211: BlockingRecv error: recv got EOF from 127.0.98.3:37211 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 19:01:15 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:15 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:15.677136 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:15.669638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:53323)
0925 19:01:15.669765 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:15.669774 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:15.669824 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:15.669952 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:15.669953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:15.670094 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:15.670149 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:15.670151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:15.670699 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:15.670700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:15.671151 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:15.671153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:15.671463 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:15.671463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:15.671514 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:15.675358 (+ 3844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:15.675363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:15.677048 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
W20260925 19:01:15.677197 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:15.669710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53323 (local address 127.0.98.62:43835)
0925 19:01:15.669797 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:15.669799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:15.669814 (+ 15us) server_negotiation.cc:407] Connection header received
0925 19:01:15.669876 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:15.669876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:15.669886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:15.669939 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:15.670199 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:15.670643 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:15.671209 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:15.671409 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:15.671451 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:15.677133 (+ 5682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53323: BlockingRecv error: recv got EOF from 127.0.98.1:53323 (error 108)
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":42,"threads_started":1}
Sep 25 19:01:15 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 19:01:15 dist-test-slave-49bz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 19:01:15.784662 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:15.780494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:48339)
0925 19:01:15.780609 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:15.780620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:15.780679 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:15.780797 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:15.780797 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:15.780943 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:15.780995 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:15.780996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:15.781599 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:15.781600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:15.782061 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:15.782062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:15.782375 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:15.782375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:15.782412 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:15.782884 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:15.782888 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:15.784572 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43835: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":50,"threads_started":1}
W20260925 19:01:15.784708 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:15.780578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48339 (local address 127.0.98.62:43835)
0925 19:01:15.780642 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:15.780644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:15.780664 (+ 20us) server_negotiation.cc:407] Connection header received
0925 19:01:15.780728 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:15.780729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:15.780739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:15.780778 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:15.781055 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:15.781533 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:15.782127 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:15.782306 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:15.782354 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:15.784667 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48339: BlockingRecv error: recv got EOF from 127.0.98.2:48339 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:01:15.815901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 9142
I20260925 19:01:15.820356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 9294
I20260925 19:01:15.824442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 9426
I20260925 19:01:15.828444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 9072
2026-09-25T19:01:15Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17322 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T19:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:01:15Z Disabled control of system clock
I20260925 19:01:15.855039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38281
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 19:01:15.934314 9622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:15.934535 9622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:15.934553 9622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:15.935680 9622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:01:15.935714 9622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:15.935726 9622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:15.935739 9622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:01:15.935751 9622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:01:15.936993 9622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:15.937165 9622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:15.937387 9622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:15.942287 9630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:15.942214 9627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:15.942281 9622 server_base.cc:1034] running on GCE node
W20260925 19:01:15.942206 9628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:15.942734 9622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:15.943024 9622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:15.944168 9622 hybrid_clock.cc:648] HybridClock initialized: now 1790362875944159 us; error 44 us; skew 500 ppm
I20260925 19:01:15.945328 9622 webserver.cc:460] Webserver started at http://127.0.98.62:44351/ using document root <none> and password file <none>
I20260925 19:01:15.945559 9622 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:15.945603 9622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:15.945748 9622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:15.946821 9622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "e4646ea38b724cb2a79da7e5bc5d55ff"
format_stamp: "Formatted at 2026-09-25 19:01:15 on dist-test-slave-49bz"
server_key: "eaa4e16555dd329aa7256979039e6ee0"
server_key_iv: "29583ffe60091a057b84126412ad6b82"
server_key_version: "encryptionkey@0"
I20260925 19:01:15.947166 9622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4646ea38b724cb2a79da7e5bc5d55ff"
format_stamp: "Formatted at 2026-09-25 19:01:15 on dist-test-slave-49bz"
server_key: "eaa4e16555dd329aa7256979039e6ee0"
server_key_iv: "29583ffe60091a057b84126412ad6b82"
server_key_version: "encryptionkey@0"
I20260925 19:01:15.948364 9622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:01:15.949005 9636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:15.949191 9622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:01:15.949271 9622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "e4646ea38b724cb2a79da7e5bc5d55ff"
format_stamp: "Formatted at 2026-09-25 19:01:15 on dist-test-slave-49bz"
server_key: "eaa4e16555dd329aa7256979039e6ee0"
server_key_iv: "29583ffe60091a057b84126412ad6b82"
server_key_version: "encryptionkey@0"
I20260925 19:01:15.949337 9622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:15.959750 9622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:15.959985 9622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:15.960112 9622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:15.964114 9622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38281
I20260925 19:01:15.964141 9688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38281 every 8 connection(s)
I20260925 19:01:15.964414 9622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:01:15.965045 9689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:01:15.967550 9689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff: Bootstrap starting.
I20260925 19:01:15.968092 9689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff: Neither blocks nor log segments found. Creating new log.
I20260925 19:01:15.968377 9689 log.cc:824] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff: Log is configured to *not* fsync() on all Append() calls
I20260925 19:01:15.968983 9689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff: No bootstrap required, opened a new log
I20260925 19:01:15.970499 9689 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4646ea38b724cb2a79da7e5bc5d55ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38281 } }
I20260925 19:01:15.970614 9689 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:01:15.970633 9689 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4646ea38b724cb2a79da7e5bc5d55ff, State: Initialized, Role: FOLLOWER
I20260925 19:01:15.970769 9689 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4646ea38b724cb2a79da7e5bc5d55ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38281 } }
I20260925 19:01:15.970827 9689 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:01:15.970846 9689 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:01:15.970909 9689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:01:15.971374 9689 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4646ea38b724cb2a79da7e5bc5d55ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38281 } }
I20260925 19:01:15.971488 9689 leader_election.cc:302] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4646ea38b724cb2a79da7e5bc5d55ff; no voters:
I20260925 19:01:15.971619 9689 leader_election.cc:288] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:01:15.971655 9692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:01:15.971808 9692 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [term 1 LEADER]: Becoming Leader. State: Replica: e4646ea38b724cb2a79da7e5bc5d55ff, State: Running, Role: LEADER
I20260925 19:01:15.971891 9689 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:01:15.971943 9692 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4646ea38b724cb2a79da7e5bc5d55ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38281 } }
I20260925 19:01:15.972357 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4646ea38b724cb2a79da7e5bc5d55ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4646ea38b724cb2a79da7e5bc5d55ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38281 } } }
I20260925 19:01:15.972433 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [sys.catalog]: This master's current role is: LEADER
I20260925 19:01:15.972618 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4646ea38b724cb2a79da7e5bc5d55ff. Latest consensus state: current_term: 1 leader_uuid: "e4646ea38b724cb2a79da7e5bc5d55ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4646ea38b724cb2a79da7e5bc5d55ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38281 } } }
I20260925 19:01:15.972848 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff [sys.catalog]: This master's current role is: LEADER
I20260925 19:01:15.972886 9701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:01:15.973372 9701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:01:15.973994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 9622
I20260925 19:01:15.974085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:01:15.974296 392 external_mini_cluster.cc:1428] Setting key c08ecb4f7ff718b08d0f435329b444ca
I20260925 19:01:15.974798 9701 catalog_manager.cc:1347] Generated new cluster ID: 44bfd49c2fdc47888ff6f4d39261fdea
I20260925 19:01:15.974881 9701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:01:15.982195 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:15.974796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.0.98.62:38281)
0925 19:01:15.978388 (+ 3592us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:15.978394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:15.978417 (+ 23us) server_negotiation.cc:407] Connection header received
0925 19:01:15.978495 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:15.978500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:15.978545 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:15.978671 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:15.978880 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:15.979704 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:15.980444 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:15.980837 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:15.980971 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:15.981591 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:15.981595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:15.981595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:01:15.981745 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:15.981773 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:01:15.981782 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:15.981847 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:15.981923 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 19:01:15.982064 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3521,"thread_start_us":67,"threads_started":1}
I20260925 19:01:15.997428 9701 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:01:15.998090 9701 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:01:16.007781 9701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4646ea38b724cb2a79da7e5bc5d55ff: Generated new TSK 0
I20260925 19:01:16.007961 9701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:01:16.014070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38281
--builtin_ntp_servers=127.0.98.20:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 19:01:16.099781 9713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:16.099961 9713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:16.099980 9713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:16.101051 9713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:16.101085 9713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:16.101107 9713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:01:16.102378 9713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:16.102591 9713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:16.102850 9713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:16.108381 9718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:16.108415 9719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.108551 9713 server_base.cc:1034] running on GCE node
W20260925 19:01:16.108624 9721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.108820 9713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:16.109051 9713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:16.110208 9713 hybrid_clock.cc:648] HybridClock initialized: now 1790362876110192 us; error 26 us; skew 500 ppm
I20260925 19:01:16.111469 9713 webserver.cc:460] Webserver started at http://127.0.98.1:41757/ using document root <none> and password file <none>
I20260925 19:01:16.111693 9713 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:16.111744 9713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:16.111860 9713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:16.113144 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "94397abcb6844cb9a037cdbb085f7090"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "6d219c2ccd4e2cac34ddf9b2034aebda"
server_key_iv: "be5060b5480106774576b28da9405bfe"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.113555 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "94397abcb6844cb9a037cdbb085f7090"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "6d219c2ccd4e2cac34ddf9b2034aebda"
server_key_iv: "be5060b5480106774576b28da9405bfe"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.115027 9713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:01:16.115816 9727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.116000 9713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:01:16.116089 9713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "94397abcb6844cb9a037cdbb085f7090"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "6d219c2ccd4e2cac34ddf9b2034aebda"
server_key_iv: "be5060b5480106774576b28da9405bfe"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.116158 9713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:16.122776 9713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:16.123016 9713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:16.123143 9713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:16.123376 9713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:16.123657 9713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:16.123690 9713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.123710 9713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:16.123723 9713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.129014 9713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45007
I20260925 19:01:16.129081 9840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45007 every 8 connection(s)
I20260925 19:01:16.129397 9713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:01:16.134174 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.130090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47641 (local address 127.0.98.62:38281)
0925 19:01:16.130150 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:16.130152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:16.130376 (+ 224us) server_negotiation.cc:407] Connection header received
0925 19:01:16.130794 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:16.130795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:16.130808 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:16.130866 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:16.131505 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.131986 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.132912 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.133101 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.133154 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:16.133870 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:16.133871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:16.133871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:01:16.133875 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:16.133887 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:01:16.133891 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:16.133942 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:16.134095 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 19:01:16.134120 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:01:16.134447 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.130024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38281 (local address 127.0.98.1:47641)
0925 19:01:16.130219 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:16.130272 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:16.130718 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:16.130915 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:16.130918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:16.130926 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:16.131434 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:16.131438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.132066 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.132067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:16.132837 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:16.132839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.133170 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.133171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:16.133284 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:16.133745 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:01:16.133745 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:16.133771 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:01:16.133790 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:16.133985 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:16.133990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:16.133993 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:01:16.134079 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:01:16.134158 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260925 19:01:16.134923 9841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38281
I20260925 19:01:16.135006 9841 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:16.135131 9841 heartbeater.cc:511] Master 127.0.98.62:38281 requested a full tablet report, sending...
I20260925 19:01:16.135634 9653 ts_manager.cc:194] Registered new tserver with Master: 94397abcb6844cb9a037cdbb085f7090 (127.0.98.1:45007)
I20260925 19:01:16.136237 9653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47641
I20260925 19:01:16.138952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 9713
I20260925 19:01:16.139024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:01:16.139173 392 external_mini_cluster.cc:1428] Setting key 470bb606e76406861ef7d3982960c1f0
I20260925 19:01:16.139981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38281
--builtin_ntp_servers=127.0.98.20:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 19:01:16.214578 9844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:16.214744 9844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:16.214762 9844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:16.215787 9844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:16.215818 9844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:16.215842 9844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:01:16.216979 9844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:16.217185 9844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:16.217438 9844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:16.222107 9852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.222146 9844 server_base.cc:1034] running on GCE node
W20260925 19:01:16.222240 9850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:16.222473 9849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.222754 9844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:16.222985 9844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:16.224138 9844 hybrid_clock.cc:648] HybridClock initialized: now 1790362876224127 us; error 23 us; skew 500 ppm
I20260925 19:01:16.225415 9844 webserver.cc:460] Webserver started at http://127.0.98.2:34369/ using document root <none> and password file <none>
I20260925 19:01:16.225646 9844 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:16.225714 9844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:16.225845 9844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:16.226909 9844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "40aa8a57fd874f3f8fa08b25fb933286"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "fd04b63b7ce193c2b4693c052ea6dede"
server_key_iv: "97208a40b08f559a5bb56184949c1d65"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.227239 9844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "40aa8a57fd874f3f8fa08b25fb933286"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "fd04b63b7ce193c2b4693c052ea6dede"
server_key_iv: "97208a40b08f559a5bb56184949c1d65"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.228433 9844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:01:16.229056 9858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.229211 9844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:01:16.229298 9844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "40aa8a57fd874f3f8fa08b25fb933286"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "fd04b63b7ce193c2b4693c052ea6dede"
server_key_iv: "97208a40b08f559a5bb56184949c1d65"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.229375 9844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:16.235955 9844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:16.236195 9844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:16.236346 9844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:16.236562 9844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:16.236826 9844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:16.236877 9844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.236908 9844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:16.236932 9844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.242141 9844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34629
I20260925 19:01:16.242187 9971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34629 every 8 connection(s)
I20260925 19:01:16.242478 9844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:01:16.243896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 9844
I20260925 19:01:16.243978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:01:16.244161 392 external_mini_cluster.cc:1428] Setting key d72e9c1156cbb9e89e43162f048cf4f4
I20260925 19:01:16.244982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38281
--builtin_ntp_servers=127.0.98.20:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 19:01:16.247593 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.243083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51705 (local address 127.0.98.62:38281)
0925 19:01:16.243157 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:16.243159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:16.243336 (+ 177us) server_negotiation.cc:407] Connection header received
0925 19:01:16.243742 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:16.243743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:16.243755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:16.243825 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:16.244567 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.245240 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.246176 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.246450 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.246532 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:16.247343 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:16.247344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:16.247344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:01:16.247349 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:16.247364 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:01:16.247368 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:16.247423 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:16.247467 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 19:01:16.247492 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:01:16.247678 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.243046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38281 (local address 127.0.98.2:51705)
0925 19:01:16.243224 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:16.243249 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:16.243648 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:16.243882 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:16.243884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:16.243892 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:16.244484 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:16.244488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.245341 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.245342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:16.246085 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:16.246087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.246539 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.246540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:16.246672 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:16.247220 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:01:16.247221 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:16.247242 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:01:16.247260 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:16.247424 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:16.247426 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:16.247427 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 19:01:16.247465 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 19:01:16.247528 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260925 19:01:16.248116 9972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38281
I20260925 19:01:16.248205 9972 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:16.248363 9972 heartbeater.cc:511] Master 127.0.98.62:38281 requested a full tablet report, sending...
I20260925 19:01:16.248700 9653 ts_manager.cc:194] Registered new tserver with Master: 40aa8a57fd874f3f8fa08b25fb933286 (127.0.98.2:34629)
I20260925 19:01:16.249042 9653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51705
W20260925 19:01:16.319463 9975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:16.319630 9975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:16.319676 9975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:16.320792 9975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:16.320830 9975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:16.320853 9975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:01:16.322028 9975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:16.322181 9975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:16.322444 9975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:16.326958 9983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:16.326962 9981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.327113 9975 server_base.cc:1034] running on GCE node
W20260925 19:01:16.327018 9980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.327368 9975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:16.327600 9975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:16.328758 9975 hybrid_clock.cc:648] HybridClock initialized: now 1790362876328743 us; error 27 us; skew 500 ppm
I20260925 19:01:16.329874 9975 webserver.cc:460] Webserver started at http://127.0.98.3:43689/ using document root <none> and password file <none>
I20260925 19:01:16.330107 9975 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:16.330174 9975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:16.330330 9975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:16.331261 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "c5feae25452b47ed8a76d80e6380e66e"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "07c96b40a740bc29cab51e20f7fe5ed5"
server_key_iv: "9948f07b1bb9654a4097f760c4e61de5"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.331672 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c5feae25452b47ed8a76d80e6380e66e"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "07c96b40a740bc29cab51e20f7fe5ed5"
server_key_iv: "9948f07b1bb9654a4097f760c4e61de5"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.332921 9975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:01:16.333631 9989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.333827 9975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:01:16.333915 9975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "c5feae25452b47ed8a76d80e6380e66e"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "07c96b40a740bc29cab51e20f7fe5ed5"
server_key_iv: "9948f07b1bb9654a4097f760c4e61de5"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.333993 9975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:16.349607 9975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:16.349838 9975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:16.349960 9975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:16.350133 9975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:16.350441 9975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:16.350492 9975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.350530 9975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:16.350555 9975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.356079 9975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46453
I20260925 19:01:16.356119 10102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46453 every 8 connection(s)
I20260925 19:01:16.356432 9975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:01:16.359687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 9975
I20260925 19:01:16.359768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:01:16.359969 392 external_mini_cluster.cc:1428] Setting key 2de3416a8d6a9603e09f340addd474ff
I20260925 19:01:16.361112 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.357040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38891 (local address 127.0.98.62:38281)
0925 19:01:16.357109 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:16.357111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:16.357270 (+ 159us) server_negotiation.cc:407] Connection header received
0925 19:01:16.357685 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:16.357687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:16.357696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:16.357750 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:16.358321 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.358786 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.359624 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.359902 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.359973 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:16.360840 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:16.360842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:16.360842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:01:16.360847 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:16.360861 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:01:16.360866 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:16.360919 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:16.360999 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 19:01:16.361028 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:01:16.361322 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.357009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38281 (local address 127.0.98.3:38891)
0925 19:01:16.357175 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:16.357202 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:16.357572 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:16.357810 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:16.357813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:16.357820 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:16.358215 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:16.358218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.358868 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.358869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:16.359551 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:16.359552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:16.359978 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:16.359978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:16.360116 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:16.360677 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:01:16.360678 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:16.360726 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:01:16.360750 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:16.360920 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:16.360922 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:16.360925 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:01:16.361002 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:01:16.361108 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260925 19:01:16.361738 10103 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38281
I20260925 19:01:16.361819 10103 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:16.361968 10103 heartbeater.cc:511] Master 127.0.98.62:38281 requested a full tablet report, sending...
I20260925 19:01:16.362277 9653 ts_manager.cc:194] Registered new tserver with Master: c5feae25452b47ed8a76d80e6380e66e (127.0.98.3:46453)
I20260925 19:01:16.362588 9653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38891
I20260925 19:01:16.371886 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 19:01:16.373627 9712 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:16.373101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.0.98.62:38281)
0925 19:01:16.373133 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:16.373134 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:16.373290 (+ 156us) server_negotiation.cc:407] Connection header received
0925 19:01:16.373351 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:16.373352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:16.373374 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:16.373429 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:16.373570 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58002: BlockingRecv error: recv got EOF from 127.0.0.1:58002 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:01:16.373713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 9713
I20260925 19:01:16.378394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 9844
I20260925 19:01:16.382161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 9975
I20260925 19:01:16.386204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 9622
2026-09-25T19:01:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (558 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T19:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:01:16Z Disabled control of system clock
I20260925 19:01:16.413173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39631
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:01:16.491320 10114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:16.491518 10114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:16.491536 10114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:16.492538 10114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:01:16.492583 10114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:16.492594 10114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:16.492604 10114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:01:16.492615 10114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:01:16.493832 10114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:16.493994 10114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:16.494186 10114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:16.498579 10122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:16.498569 10120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:16.498600 10119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.498750 10114 server_base.cc:1034] running on GCE node
I20260925 19:01:16.498934 10114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:16.499138 10114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:16.500284 10114 hybrid_clock.cc:648] HybridClock initialized: now 1790362876500260 us; error 34 us; skew 500 ppm
I20260925 19:01:16.501504 10114 webserver.cc:460] Webserver started at http://127.0.98.62:44237/ using document root <none> and password file <none>
I20260925 19:01:16.501724 10114 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:16.501766 10114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:16.501916 10114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:16.502904 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "9c2ff8b0567135e30b91329b0ac880df"
server_key_iv: "8f2a82dadf851d34934c4c4550191d8d"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.503237 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "9c2ff8b0567135e30b91329b0ac880df"
server_key_iv: "8f2a82dadf851d34934c4c4550191d8d"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.504426 10114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:01:16.505079 10128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.505228 10114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:01:16.505312 10114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "9c2ff8b0567135e30b91329b0ac880df"
server_key_iv: "8f2a82dadf851d34934c4c4550191d8d"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.505363 10114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:16.508414 10114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:16.508598 10114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:16.508685 10114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:16.511945 10114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39631
I20260925 19:01:16.511976 10180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39631 every 8 connection(s)
I20260925 19:01:16.512209 10114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:01:16.512750 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:01:16.515276 10181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf: Bootstrap starting.
I20260925 19:01:16.515820 10181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf: Neither blocks nor log segments found. Creating new log.
I20260925 19:01:16.516103 10181 log.cc:824] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf: Log is configured to *not* fsync() on all Append() calls
I20260925 19:01:16.516776 10181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf: No bootstrap required, opened a new log
I20260925 19:01:16.518343 10181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39631 } }
I20260925 19:01:16.518471 10181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:01:16.518492 10181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a43c7bcc1274ca4bd4390fc946ee3bf, State: Initialized, Role: FOLLOWER
I20260925 19:01:16.518643 10181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39631 } }
I20260925 19:01:16.518702 10181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:01:16.518754 10181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:01:16.518800 10181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:01:16.519291 10181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39631 } }
I20260925 19:01:16.519377 10181 leader_election.cc:302] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a43c7bcc1274ca4bd4390fc946ee3bf; no voters:
I20260925 19:01:16.519546 10181 leader_election.cc:288] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:01:16.519634 10184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:01:16.519760 10181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:01:16.519783 10184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [term 1 LEADER]: Becoming Leader. State: Replica: 2a43c7bcc1274ca4bd4390fc946ee3bf, State: Running, Role: LEADER
I20260925 19:01:16.519842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 10114
I20260925 19:01:16.519884 10184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39631 } }
I20260925 19:01:16.519930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:01:16.520130 392 external_mini_cluster.cc:1428] Setting key b605d29a7c5b1fc921bb18b120e2aaf5
I20260925 19:01:16.520507 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a43c7bcc1274ca4bd4390fc946ee3bf. Latest consensus state: current_term: 1 leader_uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39631 } } }
I20260925 19:01:16.520596 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [sys.catalog]: This master's current role is: LEADER
I20260925 19:01:16.520911 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a43c7bcc1274ca4bd4390fc946ee3bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39631 } } }
I20260925 19:01:16.520991 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf [sys.catalog]: This master's current role is: LEADER
W20260925 19:01:16.521502 10201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:01:16.521551 10201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:01:16.521595 10194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:01:16.521765 10194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:01:16.523226 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.520615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.0.98.62:39631)
0925 19:01:16.521787 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:16.521794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:16.521841 (+ 47us) server_negotiation.cc:407] Connection header received
0925 19:01:16.521942 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:16.521948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:16.522000 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:16.522131 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:16.522798 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:16.522802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:16.522802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:01:16.522806 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:16.522858 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:01:16.522873 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:16.522954 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:16.523016 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:01:16.523069 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095,"thread_start_us":79,"threads_started":1}
I20260925 19:01:16.523784 10194 catalog_manager.cc:1347] Generated new cluster ID: 2b81bb41368d4b3a832ee67d3b5c7598
I20260925 19:01:16.523831 10194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:01:16.530061 10194 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:01:16.530197 10194 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:01:16.533593 10194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a43c7bcc1274ca4bd4390fc946ee3bf: Generated new TSK 0
I20260925 19:01:16.533766 10194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:01:16.538365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39631
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:01:16.619123 10205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:16.619298 10205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:16.619318 10205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:16.620515 10205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:16.620568 10205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:16.620616 10205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:01:16.622005 10205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:16.622239 10205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:16.622581 10205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:16.627156 10213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:16.627172 10210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:16.627161 10211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.627276 10205 server_base.cc:1034] running on GCE node
I20260925 19:01:16.627585 10205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:16.627805 10205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:16.628953 10205 hybrid_clock.cc:648] HybridClock initialized: now 1790362876628941 us; error 28 us; skew 500 ppm
I20260925 19:01:16.630095 10205 webserver.cc:460] Webserver started at http://127.0.98.1:46451/ using document root <none> and password file <none>
I20260925 19:01:16.630388 10205 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:16.630448 10205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:16.630566 10205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:16.631482 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "348e5fa06dba415380774ab6ec2c518b"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "cc9905811618d6d03409b4e097a6e7f8"
server_key_iv: "cb6a26fa3504ef10077134b30e258aec"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.631819 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "348e5fa06dba415380774ab6ec2c518b"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "cc9905811618d6d03409b4e097a6e7f8"
server_key_iv: "cb6a26fa3504ef10077134b30e258aec"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.633113 10205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:01:16.633837 10219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.634011 10205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:01:16.634087 10205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "348e5fa06dba415380774ab6ec2c518b"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "cc9905811618d6d03409b4e097a6e7f8"
server_key_iv: "cb6a26fa3504ef10077134b30e258aec"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.634151 10205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:16.637287 10205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:16.637552 10205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:16.637698 10205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:16.637917 10205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:16.638203 10205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:16.638233 10205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.638302 10205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:16.638329 10205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.644973 10205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37869
I20260925 19:01:16.645051 10332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37869 every 8 connection(s)
I20260925 19:01:16.645339 10205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:01:16.648151 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.646001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36189 (local address 127.0.98.62:39631)
0925 19:01:16.646039 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:16.646042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:16.646303 (+ 261us) server_negotiation.cc:407] Connection header received
0925 19:01:16.646737 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:16.646738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:16.646745 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:16.646813 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:16.647914 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:16.647915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:16.647916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:01:16.647920 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:16.647936 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:01:16.647941 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:16.648015 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:16.648067 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 19:01:16.648081 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:01:16.648274 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.645998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39631 (local address 127.0.98.1:36189)
0925 19:01:16.646157 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:16.646186 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:16.646639 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:16.646857 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:16.646859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:16.646872 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:16.647771 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:01:16.647772 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:16.647796 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:01:16.647821 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:16.648012 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:16.648014 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:16.648016 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:01:16.648058 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 19:01:16.648100 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260925 19:01:16.648737 10333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39631
I20260925 19:01:16.648837 10333 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:16.648988 10333 heartbeater.cc:511] Master 127.0.98.62:39631 requested a full tablet report, sending...
I20260925 19:01:16.649399 10145 ts_manager.cc:194] Registered new tserver with Master: 348e5fa06dba415380774ab6ec2c518b (127.0.98.1:37869)
I20260925 19:01:16.652920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 10205
I20260925 19:01:16.652985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:01:16.653169 392 external_mini_cluster.cc:1428] Setting key e6b32fab3c32fcfa1e239ecabd8ccdd2
I20260925 19:01:16.653995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39631
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:01:16.728708 10336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:16.728909 10336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:16.728940 10336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:16.730079 10336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:16.730139 10336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:16.730180 10336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:01:16.731513 10336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:16.731786 10336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:16.732069 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:16.736706 10344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:16.736753 10341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:16.736929 10342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.736884 10336 server_base.cc:1034] running on GCE node
I20260925 19:01:16.737140 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:16.737443 10336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:16.738598 10336 hybrid_clock.cc:648] HybridClock initialized: now 1790362876738573 us; error 42 us; skew 500 ppm
I20260925 19:01:16.739694 10336 webserver.cc:460] Webserver started at http://127.0.98.2:40697/ using document root <none> and password file <none>
I20260925 19:01:16.739931 10336 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:16.739981 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:16.740118 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:16.741118 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "3c416665266f4469b5134d39bb56a0d0"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "8ef4abd2d9b776534558846a1ee697d6"
server_key_iv: "aad9f7bbb86f2e4e7a422911d6501c4d"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.741436 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3c416665266f4469b5134d39bb56a0d0"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "8ef4abd2d9b776534558846a1ee697d6"
server_key_iv: "aad9f7bbb86f2e4e7a422911d6501c4d"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.742717 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:01:16.743383 10350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.743558 10336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:01:16.743638 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "3c416665266f4469b5134d39bb56a0d0"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "8ef4abd2d9b776534558846a1ee697d6"
server_key_iv: "aad9f7bbb86f2e4e7a422911d6501c4d"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.743701 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:16.746814 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:16.747048 10336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:16.747172 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:16.747365 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:16.747634 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:16.747684 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.747722 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:16.747747 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.753170 10336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40005
I20260925 19:01:16.753201 10463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40005 every 8 connection(s)
I20260925 19:01:16.753468 10336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:01:16.756265 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.754150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53449 (local address 127.0.98.62:39631)
0925 19:01:16.754224 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:16.754226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:16.754408 (+ 182us) server_negotiation.cc:407] Connection header received
0925 19:01:16.754821 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:16.754822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:16.754828 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:16.754890 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:16.756034 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:16.756035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:16.756035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:01:16.756039 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:16.756055 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:01:16.756059 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:16.756108 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:16.756187 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 19:01:16.756200 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:01:16.756467 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.754122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39631 (local address 127.0.98.2:53449)
0925 19:01:16.754308 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:16.754335 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:16.754730 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:16.754951 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:16.754959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:16.754967 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:16.755886 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:01:16.755887 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:16.755912 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:01:16.755936 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:16.756103 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:16.756107 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:16.756110 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:01:16.756196 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:01:16.756250 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260925 19:01:16.756908 10464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39631
I20260925 19:01:16.756973 10464 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:16.757114 10464 heartbeater.cc:511] Master 127.0.98.62:39631 requested a full tablet report, sending...
I20260925 19:01:16.757426 10145 ts_manager.cc:194] Registered new tserver with Master: 3c416665266f4469b5134d39bb56a0d0 (127.0.98.2:40005)
I20260925 19:01:16.757987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 10336
I20260925 19:01:16.758090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:01:16.758284 392 external_mini_cluster.cc:1428] Setting key a4de81f8f39d5c796f72ae4034ccbdfc
I20260925 19:01:16.758998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39631
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 19:01:16.834252 10467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:16.834496 10467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:16.834533 10467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:16.835695 10467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:16.835758 10467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:16.835803 10467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:01:16.837250 10467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:16.837522 10467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:16.837862 10467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:16.842476 10475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:16.842460 10473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.842628 10467 server_base.cc:1034] running on GCE node
W20260925 19:01:16.842460 10472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:16.842836 10467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:16.843062 10467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:16.844198 10467 hybrid_clock.cc:648] HybridClock initialized: now 1790362876844188 us; error 26 us; skew 500 ppm
I20260925 19:01:16.845422 10467 webserver.cc:460] Webserver started at http://127.0.98.3:33175/ using document root <none> and password file <none>
I20260925 19:01:16.845654 10467 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:16.845701 10467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:16.845861 10467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:16.846910 10467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "d0ef77320c364ab786c642b8ea6fba3b"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "9c4c1489c6aa8ea11c1f47240b14a232"
server_key_iv: "71687bb995694e92dcf81028adb0a271"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.847258 10467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d0ef77320c364ab786c642b8ea6fba3b"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "9c4c1489c6aa8ea11c1f47240b14a232"
server_key_iv: "71687bb995694e92dcf81028adb0a271"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.848507 10467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:01:16.849186 10481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.849339 10467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:01:16.849437 10467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "d0ef77320c364ab786c642b8ea6fba3b"
format_stamp: "Formatted at 2026-09-25 19:01:16 on dist-test-slave-49bz"
server_key: "9c4c1489c6aa8ea11c1f47240b14a232"
server_key_iv: "71687bb995694e92dcf81028adb0a271"
server_key_version: "encryptionkey@0"
I20260925 19:01:16.849516 10467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:16.852612 10467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:16.852852 10467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:16.852982 10467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:16.853163 10467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:16.853428 10467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:16.853477 10467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.853513 10467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:16.853538 10467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:16.860332 10467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33361
I20260925 19:01:16.860381 10594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33361 every 8 connection(s)
I20260925 19:01:16.860747 10467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:01:16.863255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 10467
I20260925 19:01:16.863332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:01:16.863533 392 external_mini_cluster.cc:1428] Setting key b6663ea3ec80a48b36356d0e213e8818
I20260925 19:01:16.864673 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.861758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42901 (local address 127.0.98.62:39631)
0925 19:01:16.861823 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:16.861825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:16.862123 (+ 298us) server_negotiation.cc:407] Connection header received
0925 19:01:16.862775 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:16.862777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:16.862784 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:16.862839 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:16.864413 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:16.864414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:16.864414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:01:16.864418 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:16.864433 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:01:16.864437 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:16.864492 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:16.864587 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 19:01:16.864603 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:01:16.864840 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:16.861755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39631 (local address 127.0.98.3:42901)
0925 19:01:16.861954 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:16.862006 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:16.862662 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:16.862982 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:16.862986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:16.862997 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:16.864253 (+ 1256us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:01:16.864254 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:16.864291 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:01:16.864325 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:16.864503 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:16.864509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:16.864511 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:01:16.864584 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:01:16.864647 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260925 19:01:16.865329 10595 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39631
I20260925 19:01:16.865432 10595 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:16.865567 10595 heartbeater.cc:511] Master 127.0.98.62:39631 requested a full tablet report, sending...
I20260925 19:01:16.865876 10145 ts_manager.cc:194] Registered new tserver with Master: d0ef77320c364ab786c642b8ea6fba3b (127.0.98.3:33361)
I20260925 19:01:16.875169 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 19:01:16.877142 10195 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:01:16.876664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.0.98.62:39631)
0925 19:01:16.876699 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:16.876701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:16.876823 (+ 122us) server_negotiation.cc:407] Connection header received
0925 19:01:16.876900 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:16.876901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:16.876906 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:16.876959 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:16.877084 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45536: BlockingRecv error: recv got EOF from 127.0.0.1:45536 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:01:16.877215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 10205
I20260925 19:01:16.884814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 10336
I20260925 19:01:16.889715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 10467
I20260925 19:01:16.894464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 10114
2026-09-25T19:01:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (507 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:01:16 dist-test-slave-49bz krb5kdc[10605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:01:16 dist-test-slave-49bz krb5kdc[10605](info): set up 2 sockets
Sep 25 19:01:16 dist-test-slave-49bz krb5kdc[10605](info): commencing operation
krb5kdc: starting...
W20260925 19:01:17.874509 392 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:01:17 dist-test-slave-49bz krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:01:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:01:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:01:17.982404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46009
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:01:18.056769 10618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:18.056983 10618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:18.057015 10618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:18.058051 10618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:01:18.058116 10618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:18.058137 10618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:18.058156 10618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:01:18.058183 10618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:01:18.059477 10618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:18.059710 10618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:18.059908 10618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:18.064245 10623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:18.064266 10624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:18.064266 10626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:18.064440 10618 server_base.cc:1034] running on GCE node
I20260925 19:01:18.064651 10618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:18.064961 10618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:18.066133 10618 hybrid_clock.cc:648] HybridClock initialized: now 1790362878066127 us; error 35 us; skew 500 ppm
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:18.068368 10618 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:01:18.068796 10618 webserver.cc:460] Webserver started at http://127.0.98.62:39375/ using document root <none> and password file <none>
I20260925 19:01:18.068975 10618 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:18.069012 10618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:18.069109 10618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:18.069971 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "226e6a833ccb4cd89ef707233b411fcd"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "b37b2b3260347bcef7f21e9e70a55f54"
server_key_iv: "836881af1714ec544531ca87d1ffe113"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.070240 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "226e6a833ccb4cd89ef707233b411fcd"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "b37b2b3260347bcef7f21e9e70a55f54"
server_key_iv: "836881af1714ec544531ca87d1ffe113"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.071496 10618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:01:18.072182 10633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:18.072397 10618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:01:18.072470 10618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "226e6a833ccb4cd89ef707233b411fcd"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "b37b2b3260347bcef7f21e9e70a55f54"
server_key_iv: "836881af1714ec544531ca87d1ffe113"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.072561 10618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:18.081712 10618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:18.083453 10618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:18.083608 10618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:18.087366 10618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46009
I20260925 19:01:18.087531 10685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46009 every 8 connection(s)
I20260925 19:01:18.087714 10618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:01:18.088544 10686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:01:18.090576 10686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd: Bootstrap starting.
I20260925 19:01:18.091046 10686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd: Neither blocks nor log segments found. Creating new log.
I20260925 19:01:18.091245 10686 log.cc:824] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:01:18.091789 10686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd: No bootstrap required, opened a new log
I20260925 19:01:18.093291 10686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226e6a833ccb4cd89ef707233b411fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46009 } }
I20260925 19:01:18.093420 10686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:01:18.093453 10686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226e6a833ccb4cd89ef707233b411fcd, State: Initialized, Role: FOLLOWER
I20260925 19:01:18.093556 10686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226e6a833ccb4cd89ef707233b411fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46009 } }
I20260925 19:01:18.093634 10686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:01:18.093669 10686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:01:18.093726 10686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:01:18.094463 10686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226e6a833ccb4cd89ef707233b411fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46009 } }
I20260925 19:01:18.094574 10686 leader_election.cc:302] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 226e6a833ccb4cd89ef707233b411fcd; no voters:
I20260925 19:01:18.094739 10686 leader_election.cc:288] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:01:18.094786 10689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:01:18.094944 10689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [term 1 LEADER]: Becoming Leader. State: Replica: 226e6a833ccb4cd89ef707233b411fcd, State: Running, Role: LEADER
I20260925 19:01:18.094970 10686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:01:18.095057 10689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226e6a833ccb4cd89ef707233b411fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46009 } }
I20260925 19:01:18.095479 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "226e6a833ccb4cd89ef707233b411fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226e6a833ccb4cd89ef707233b411fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46009 } } }
I20260925 19:01:18.095494 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 226e6a833ccb4cd89ef707233b411fcd. Latest consensus state: current_term: 1 leader_uuid: "226e6a833ccb4cd89ef707233b411fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226e6a833ccb4cd89ef707233b411fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46009 } } }
I20260925 19:01:18.095674 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [sys.catalog]: This master's current role is: LEADER
I20260925 19:01:18.095877 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd [sys.catalog]: This master's current role is: LEADER
I20260925 19:01:18.095973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 10618
I20260925 19:01:18.096050 10698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:01:18.096079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:01:18.096264 392 external_mini_cluster.cc:1428] Setting key 995101184a1e51e4ddd834b45a8f757e
I20260925 19:01:18.096661 10698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:01:18.098031 10698 catalog_manager.cc:1347] Generated new cluster ID: f1494dbf6fc9421984e2307d89098b86
I20260925 19:01:18.098085 10698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:01:18.108343 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.096703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.0.98.62:46009)
0925 19:01:18.098358 (+ 1655us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.098365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.098399 (+ 34us) server_negotiation.cc:407] Connection header received
0925 19:01:18.098492 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.098497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.098540 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.098658 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:18.099362 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.100688 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.101704 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.102223 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.102321 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.104832 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:18.104836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:18.104836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:18.104849 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:18.106848 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:18.107204 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:18.107207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:18.107207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:18.107304 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:18.107502 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:18.107502 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:18.107503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:18.107771 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:18.107903 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.108007 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.108146 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1613,"thread_start_us":75,"threads_started":1}
I20260925 19:01:18.110188 10698 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:01:18.110855 10698 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:01:18.115461 10698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd: Generated new TSK 0
I20260925 19:01:18.115605 10698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:01:18.142555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46009
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:01:18.218505 10712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:18.218708 10712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:18.218743 10712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:18.219739 10712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:18.219796 10712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:18.219834 10712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:01:18.221199 10712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:18.221437 10712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:18.221660 10712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:18.226120 10720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:18.226176 10718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:18.226172 10717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:18.226377 10712 server_base.cc:1034] running on GCE node
I20260925 19:01:18.226603 10712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:18.226831 10712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:18.228018 10712 hybrid_clock.cc:648] HybridClock initialized: now 1790362878228002 us; error 29 us; skew 500 ppm
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:18.230351 10712 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:01:18.230738 10712 webserver.cc:460] Webserver started at http://127.0.98.1:43387/ using document root <none> and password file <none>
I20260925 19:01:18.230892 10712 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:18.230934 10712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:18.231010 10712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:18.231827 10712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "c94b7f8cb3294fb8872f2b69b95a07be"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "ed5afc598de37cf891ffd837008fc913"
server_key_iv: "bba3e5e2d6f6667d8b6e15dfde380d8b"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.232097 10712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c94b7f8cb3294fb8872f2b69b95a07be"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "ed5afc598de37cf891ffd837008fc913"
server_key_iv: "bba3e5e2d6f6667d8b6e15dfde380d8b"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.233179 10712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:01:18.233767 10727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:18.233937 10712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:01:18.234014 10712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "c94b7f8cb3294fb8872f2b69b95a07be"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "ed5afc598de37cf891ffd837008fc913"
server_key_iv: "bba3e5e2d6f6667d8b6e15dfde380d8b"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.234061 10712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:18.250424 10712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:18.251917 10712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:18.252010 10712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:18.252153 10712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:18.252360 10712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:18.252388 10712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:18.252405 10712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:18.252416 10712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:18.257627 10712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38397
I20260925 19:01:18.257702 10840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38397 every 8 connection(s)
I20260925 19:01:18.257984 10712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:01:18.266737 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.258742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41801 (local address 127.0.98.62:46009)
0925 19:01:18.258787 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.258788 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.258995 (+ 207us) server_negotiation.cc:407] Connection header received
0925 19:01:18.259528 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.259529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.259545 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.259598 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:18.260500 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.261012 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.261981 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.262181 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.262312 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.264225 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:18.264226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:18.264226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:18.264234 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:18.265376 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:18.265769 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:18.265770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:18.265770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:18.265825 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:18.266062 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:18.266063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:18.266064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:18.266173 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:18.266249 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.266546 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.266627 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:01:18.266870 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.258707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46009 (local address 127.0.98.1:41801)
0925 19:01:18.258870 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:18.258912 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:18.259402 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:18.259660 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:18.259664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:18.260209 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:18.260400 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.260404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.261107 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.261108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.261890 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.261892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.262333 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.262333 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.262456 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.262876 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:18.262884 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:18.264133 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:18.265471 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:18.265478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:18.265479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:18.265685 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:18.265889 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:18.265889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:18.265890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:18.265993 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:18.266240 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:18.266246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:18.266405 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 19:01:18.266526 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 19:01:18.266661 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260925 19:01:18.266875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 10712
I20260925 19:01:18.266968 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:01:18.267176 392 external_mini_cluster.cc:1428] Setting key c770d673a7c956d2bbd5f21d2aa5e339
I20260925 19:01:18.267304 10841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46009
I20260925 19:01:18.267380 10841 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:18.267572 10841 heartbeater.cc:511] Master 127.0.98.62:46009 requested a full tablet report, sending...
I20260925 19:01:18.268078 10650 ts_manager.cc:194] Registered new tserver with Master: c94b7f8cb3294fb8872f2b69b95a07be (127.0.98.1:38397)
I20260925 19:01:18.268626 10650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41801
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:01:18.288442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46009
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:01:18.363842 10846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:18.364012 10846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:18.364033 10846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:18.365097 10846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:18.365135 10846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:18.365183 10846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:01:18.366673 10846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:18.366887 10846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:18.367096 10846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:18.371522 10851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:18.371505 10852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:18.371683 10854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:18.371713 10846 server_base.cc:1034] running on GCE node
I20260925 19:01:18.371923 10846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:18.372170 10846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:18.373303 10846 hybrid_clock.cc:648] HybridClock initialized: now 1790362878373286 us; error 33 us; skew 500 ppm
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:18.375624 10846 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:01:18.376012 10846 webserver.cc:460] Webserver started at http://127.0.98.2:45577/ using document root <none> and password file <none>
I20260925 19:01:18.376271 10846 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:18.376346 10846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:18.376509 10846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:18.377450 10846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "c4ce859ccc7643cba09b98e565b9d7b2"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "085a001ef30c4aa39289b8b95038b66f"
server_key_iv: "a18601728401d6972d54b447977ef73a"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.377820 10846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c4ce859ccc7643cba09b98e565b9d7b2"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "085a001ef30c4aa39289b8b95038b66f"
server_key_iv: "a18601728401d6972d54b447977ef73a"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.379110 10846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:01:18.379731 10861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:18.379873 10846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:01:18.379957 10846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "c4ce859ccc7643cba09b98e565b9d7b2"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "085a001ef30c4aa39289b8b95038b66f"
server_key_iv: "a18601728401d6972d54b447977ef73a"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.380029 10846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:18.394044 10846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:18.396301 10846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:18.396464 10846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:18.396680 10846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:18.397007 10846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:18.397042 10846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:18.397063 10846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:18.397105 10846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:18.402796 10846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35835
I20260925 19:01:18.402843 10974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35835 every 8 connection(s)
I20260925 19:01:18.403158 10846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:01:18.411706 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.403899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60831 (local address 127.0.98.62:46009)
0925 19:01:18.403948 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.403950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.404168 (+ 218us) server_negotiation.cc:407] Connection header received
0925 19:01:18.404609 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.404611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.404619 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.404665 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:18.405532 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.406009 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.407009 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.407185 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.407237 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.409175 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:18.409177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:18.409177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:18.409182 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:18.410657 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:18.410995 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:18.410995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:18.410995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:18.411042 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:18.411238 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:18.411239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:18.411239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:18.411331 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:18.411378 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.411593 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.411648 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:01:18.411877 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.403886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46009 (local address 127.0.98.2:60831)
0925 19:01:18.404055 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:18.404088 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:18.404533 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:18.404720 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:18.404723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:18.405272 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:18.405460 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.405465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.406084 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.406084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.406940 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.406943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.407255 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.407255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.407348 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.407763 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:18.407771 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:18.409097 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:18.410722 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:18.410727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:18.410729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:18.410931 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:18.411097 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:18.411098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:18.411098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:18.411187 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:18.411383 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:18.411387 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:18.411485 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 19:01:18.411573 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 19:01:18.411675 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260925 19:01:18.412269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 10846
I20260925 19:01:18.412360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:01:18.412367 10975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46009
I20260925 19:01:18.412444 10975 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:18.412539 392 external_mini_cluster.cc:1428] Setting key 22702a34d9266089b8a392937a129c45
I20260925 19:01:18.412622 10975 heartbeater.cc:511] Master 127.0.98.62:46009 requested a full tablet report, sending...
I20260925 19:01:18.413028 10649 ts_manager.cc:194] Registered new tserver with Master: c4ce859ccc7643cba09b98e565b9d7b2 (127.0.98.2:35835)
I20260925 19:01:18.413511 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60831
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:01:18.434607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46009
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:01:18.510015 10980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:18.510185 10980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:18.510206 10980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:18.511266 10980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:18.511307 10980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:18.511368 10980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:01:18.512814 10980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:18.513036 10980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:18.513299 10980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:18.517863 10988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:18.517846 10986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:18.517885 10985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:18.518079 10980 server_base.cc:1034] running on GCE node
I20260925 19:01:18.518301 10980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:18.518509 10980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:18.519631 10980 hybrid_clock.cc:648] HybridClock initialized: now 1790362878519620 us; error 29 us; skew 500 ppm
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:18.521919 10980 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:01:18.522315 10980 webserver.cc:460] Webserver started at http://127.0.98.3:35647/ using document root <none> and password file <none>
I20260925 19:01:18.522540 10980 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:18.522600 10980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:18.522734 10980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:01:18.523640 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "60733d46f7c941a48f4144a190933f42"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "ff1b38c178063e29a08f159c81141cae"
server_key_iv: "5d2a7f087ec6a45672ee0cb0b44e91cf"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.523964 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "60733d46f7c941a48f4144a190933f42"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "ff1b38c178063e29a08f159c81141cae"
server_key_iv: "5d2a7f087ec6a45672ee0cb0b44e91cf"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.525127 10980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:01:18.525835 10995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:18.526005 10980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:01:18.526098 10980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "60733d46f7c941a48f4144a190933f42"
format_stamp: "Formatted at 2026-09-25 19:01:18 on dist-test-slave-49bz"
server_key: "ff1b38c178063e29a08f159c81141cae"
server_key_iv: "5d2a7f087ec6a45672ee0cb0b44e91cf"
server_key_version: "encryptionkey@0"
I20260925 19:01:18.526180 10980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:18.543040 10980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:18.545161 10980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:18.545328 10980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:18.545547 10980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:18.545830 10980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:18.545881 10980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:18.545912 10980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:18.545943 10980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:18.551563 10980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33637
I20260925 19:01:18.551633 11108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33637 every 8 connection(s)
I20260925 19:01:18.551920 10980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:01:18.559710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 10980
I20260925 19:01:18.559809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:01:18.560061 392 external_mini_cluster.cc:1428] Setting key d53112eb522c14038aa53fb6ab3e3684
I20260925 19:01:18.560508 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.552666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50499 (local address 127.0.98.62:46009)
0925 19:01:18.552712 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.552715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.552920 (+ 205us) server_negotiation.cc:407] Connection header received
0925 19:01:18.553331 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.553332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.553341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.553386 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:18.554239 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.554743 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.555729 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.555939 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.556026 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.557980 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:18.557982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:18.557982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:18.557988 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:18.559133 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:18.559473 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:18.559474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:18.559474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:18.559517 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:18.559798 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:18.559799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:18.559800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:18.559961 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:18.560012 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.560327 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.560421 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:01:18.560668 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.552648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46009 (local address 127.0.98.3:50499)
0925 19:01:18.552804 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:18.552835 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:18.553258 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:18.553441 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:18.553444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:18.553950 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:18.554111 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.554132 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.554847 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.554848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.555633 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.555635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.556034 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.556036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.556153 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.556583 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:18.556591 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:18.557895 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:18.559205 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:18.559209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:18.559211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:18.559412 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:18.559589 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:18.559590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:18.559590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:18.559716 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:18.560018 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:18.560024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:18.560173 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 19:01:18.560300 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 19:01:18.560456 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260925 19:01:18.561162 11109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46009
I20260925 19:01:18.561237 11109 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:18.561419 11109 heartbeater.cc:511] Master 127.0.98.62:46009 requested a full tablet report, sending...
I20260925 19:01:18.561784 10649 ts_manager.cc:194] Registered new tserver with Master: 60733d46f7c941a48f4144a190933f42 (127.0.98.3:33637)
I20260925 19:01:18.562115 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50499
I20260925 19:01:18.571542 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:01:18.577925 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.572639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.0.98.62:46009)
0925 19:01:18.572676 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.572678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.572767 (+ 89us) server_negotiation.cc:407] Connection header received
0925 19:01:18.572823 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.572824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.572830 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.572865 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:18.573339 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.573739 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.574443 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.574658 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.574706 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.575724 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:18.575726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:18.575726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:18.575739 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:18.577069 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:18.577364 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:18.577365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:18.577365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:18.577411 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:18.577566 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:18.577566 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:18.577567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:18.577675 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:18.577741 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.577820 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.577868 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 19:01:18.579559 10649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:01:18.581008 10649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:01:18.586966 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.583033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33637 (local address 127.0.0.1:49980)
0925 19:01:18.583679 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:18.583688 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:18.583741 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:18.583958 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:18.583959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:18.583961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:01:18.584049 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.584051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.585191 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.585192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.585991 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.585994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.586719 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.586720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.586788 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.586800 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 19:01:18.586859 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 19:01:18.586902 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260925 19:01:18.586974 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.583197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49980 (local address 127.0.98.3:33637)
0925 19:01:18.583333 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.583337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.583748 (+ 411us) server_negotiation.cc:407] Connection header received
0925 19:01:18.583809 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.583812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.583826 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.583893 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:01:18.584120 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.585101 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.586075 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.586643 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.586718 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.586768 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.586869 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.586909 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260925 19:01:18.587205 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.583010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35835 (local address 127.0.0.1:48256)
0925 19:01:18.583155 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:18.583188 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:18.583247 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:18.583662 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:18.583665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:18.583667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:01:18.583724 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.583726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.584978 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.584979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.586041 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.586043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.586988 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.586989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.587044 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.587047 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:01:18.587091 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 19:01:18.587133 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":24,"thread_start_us":55,"threads_started":1}
I20260925 19:01:18.587208 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.583013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38397 (local address 127.0.0.1:40574)
0925 19:01:18.583242 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:18.583250 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:18.583289 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:18.583792 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:18.583793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:18.583794 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:01:18.583825 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.583826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.584811 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.584812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.586178 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.586180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.587030 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.587031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.587088 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.587091 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:01:18.587131 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 19:01:18.587155 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":3,"thread_start_us":47,"threads_started":1}
I20260925 19:01:18.587389 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.583230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.0.98.1:38397)
0925 19:01:18.583435 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.583439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.583446 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:01:18.583478 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.583480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.583492 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.583593 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:01:18.583892 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.584717 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.586274 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.586913 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.586964 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.587005 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.587239 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.587283 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260925 19:01:18.587512 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.583096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.0.98.2:35835)
0925 19:01:18.583414 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.583417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.583427 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:01:18.583467 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.583469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.583483 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.583571 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:01:18.583811 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.584872 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.586121 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.586842 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.586907 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.586956 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.587388 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.587431 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":66,"threads_started":1}
I20260925 19:01:18.588004 10775 tablet_service.cc:1467] Processing CreateTablet for tablet a36ad6bb838f46c6b983052b8646abce (DEFAULT_TABLE table=test-table [id=7300be748ee042b982433d4effd7c864]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:01:18.588068 10909 tablet_service.cc:1467] Processing CreateTablet for tablet a36ad6bb838f46c6b983052b8646abce (DEFAULT_TABLE table=test-table [id=7300be748ee042b982433d4effd7c864]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:01:18.588127 11043 tablet_service.cc:1467] Processing CreateTablet for tablet a36ad6bb838f46c6b983052b8646abce (DEFAULT_TABLE table=test-table [id=7300be748ee042b982433d4effd7c864]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:01:18.588340 10775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36ad6bb838f46c6b983052b8646abce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:01:18.588340 10909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36ad6bb838f46c6b983052b8646abce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:01:18.588431 11043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36ad6bb838f46c6b983052b8646abce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:01:18.590839 11123 tablet_bootstrap.cc:492] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: Bootstrap starting.
I20260925 19:01:18.591428 11125 tablet_bootstrap.cc:492] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be: Bootstrap starting.
I20260925 19:01:18.591632 11123 tablet_bootstrap.cc:654] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: Neither blocks nor log segments found. Creating new log.
I20260925 19:01:18.592127 11125 tablet_bootstrap.cc:654] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be: Neither blocks nor log segments found. Creating new log.
I20260925 19:01:18.592216 11123 log.cc:824] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: Log is configured to *not* fsync() on all Append() calls
I20260925 19:01:18.592638 11125 log.cc:824] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be: Log is configured to *not* fsync() on all Append() calls
I20260925 19:01:18.592973 11124 tablet_bootstrap.cc:492] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: Bootstrap starting.
I20260925 19:01:18.593009 11123 tablet_bootstrap.cc:492] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: No bootstrap required, opened a new log
I20260925 19:01:18.593061 11123 ts_tablet_manager.cc:1397] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:01:18.593395 11125 tablet_bootstrap.cc:492] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be: No bootstrap required, opened a new log
I20260925 19:01:18.593472 11125 ts_tablet_manager.cc:1397] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:01:18.593585 11124 tablet_bootstrap.cc:654] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: Neither blocks nor log segments found. Creating new log.
I20260925 19:01:18.594089 11124 log.cc:824] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:01:18.594875 11124 tablet_bootstrap.cc:492] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: No bootstrap required, opened a new log
I20260925 19:01:18.594941 11124 ts_tablet_manager.cc:1397] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:01:18.595379 11125 raft_consensus.cc:348] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 } } peers { permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 } } peers { permanent_uuid: "60733d46f7c941a48f4144a190933f42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33637 } }
I20260925 19:01:18.595470 11123 raft_consensus.cc:348] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 } } peers { permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 } } peers { permanent_uuid: "60733d46f7c941a48f4144a190933f42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33637 } }
I20260925 19:01:18.595552 11125 raft_consensus.cc:374] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:01:18.595629 11123 raft_consensus.cc:374] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:01:18.595640 11125 raft_consensus.cc:729] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94b7f8cb3294fb8872f2b69b95a07be, State: Initialized, Role: FOLLOWER
I20260925 19:01:18.595665 11123 raft_consensus.cc:729] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60733d46f7c941a48f4144a190933f42, State: Initialized, Role: FOLLOWER
I20260925 19:01:18.595793 11125 consensus_queue.cc:260] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 } } peers { permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 } } peers { permanent_uuid: "60733d46f7c941a48f4144a190933f42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33637 } }
I20260925 19:01:18.595803 11123 consensus_queue.cc:260] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 } } peers { permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 } } peers { permanent_uuid: "60733d46f7c941a48f4144a190933f42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33637 } }
I20260925 19:01:18.596098 11123 ts_tablet_manager.cc:1428] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:01:18.596132 11125 ts_tablet_manager.cc:1428] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:01:18.596244 11109 heartbeater.cc:503] Master 127.0.98.62:46009 was elected leader, sending a full tablet report...
I20260925 19:01:18.596246 10841 heartbeater.cc:503] Master 127.0.98.62:46009 was elected leader, sending a full tablet report...
I20260925 19:01:18.596964 11124 raft_consensus.cc:348] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 } } peers { permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 } } peers { permanent_uuid: "60733d46f7c941a48f4144a190933f42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33637 } }
I20260925 19:01:18.597124 11124 raft_consensus.cc:374] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:01:18.597162 11124 raft_consensus.cc:729] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ce859ccc7643cba09b98e565b9d7b2, State: Initialized, Role: FOLLOWER
I20260925 19:01:18.597260 11124 consensus_queue.cc:260] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 } } peers { permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 } } peers { permanent_uuid: "60733d46f7c941a48f4144a190933f42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33637 } }
I20260925 19:01:18.597496 11124 ts_tablet_manager.cc:1428] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:01:18.597611 10975 heartbeater.cc:503] Master 127.0.98.62:46009 was elected leader, sending a full tablet report...
W20260925 19:01:18.653709 10976 tablet.cc:2369] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:01:18.668262 11130 raft_consensus.cc:482] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:01:18.668356 11130 raft_consensus.cc:504] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 } } peers { permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 } } peers { permanent_uuid: "60733d46f7c941a48f4144a190933f42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33637 } }
I20260925 19:01:18.668669 11130 leader_election.cc:288] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c94b7f8cb3294fb8872f2b69b95a07be (127.0.98.1:38397), c4ce859ccc7643cba09b98e565b9d7b2 (127.0.98.2:35835)
I20260925 19:01:18.672376 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.668925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60385 (local address 127.0.98.2:35835)
0925 19:01:18.668980 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.668982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.669092 (+ 110us) server_negotiation.cc:407] Connection header received
0925 19:01:18.669117 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.669118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.669128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.669192 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:01:18.669322 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.669982 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.671411 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.672013 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.672090 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.672099 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.672269 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.672307 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:01:18.672376 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.668868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43477 (local address 127.0.98.1:38397)
0925 19:01:18.668912 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.668914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.668923 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:01:18.668954 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.668955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.668968 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.669025 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:01:18.669188 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.669944 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.671414 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.672013 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.672090 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.672099 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.672269 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.672307 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:01:18.672417 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.668798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38397 (local address 127.0.98.3:43477)
0925 19:01:18.668842 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:18.668854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:18.668897 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:18.669032 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:18.669033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:18.669036 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:01:18.669113 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.669115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.670037 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.670037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.671327 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.671330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.672094 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.672095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.672189 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.672191 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:01:18.672249 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:01:18.672304 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":91}
I20260925 19:01:18.672423 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.668831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35835 (local address 127.0.98.3:60385)
0925 19:01:18.669011 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:18.669022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:18.669070 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:18.669207 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:18.669207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:18.669209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:01:18.669260 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.669261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.670045 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.670046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.671340 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:18.671342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.672094 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.672095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:18.672186 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.672190 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:01:18.672249 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 19:01:18.672303 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":62,"thread_start_us":109,"threads_started":1}
I20260925 19:01:18.672797 10795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36ad6bb838f46c6b983052b8646abce" candidate_uuid: "60733d46f7c941a48f4144a190933f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" is_pre_election: true
I20260925 19:01:18.672948 10795 raft_consensus.cc:2454] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60733d46f7c941a48f4144a190933f42 in term 0.
I20260925 19:01:18.672966 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36ad6bb838f46c6b983052b8646abce" candidate_uuid: "60733d46f7c941a48f4144a190933f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" is_pre_election: true
I20260925 19:01:18.673111 10929 raft_consensus.cc:2454] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60733d46f7c941a48f4144a190933f42 in term 0.
I20260925 19:01:18.673166 10997 leader_election.cc:302] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60733d46f7c941a48f4144a190933f42, c94b7f8cb3294fb8872f2b69b95a07be; no voters:
I20260925 19:01:18.673296 11130 raft_consensus.cc:2781] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:01:18.673341 11130 raft_consensus.cc:482] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:01:18.673368 11130 raft_consensus.cc:3037] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:01:18.674013 11130 raft_consensus.cc:504] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 } } peers { permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 } } peers { permanent_uuid: "60733d46f7c941a48f4144a190933f42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33637 } }
I20260925 19:01:18.674147 11130 leader_election.cc:288] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [CANDIDATE]: Term 1 election: Requested vote from peers c94b7f8cb3294fb8872f2b69b95a07be (127.0.98.1:38397), c4ce859ccc7643cba09b98e565b9d7b2 (127.0.98.2:35835)
I20260925 19:01:18.674290 10795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36ad6bb838f46c6b983052b8646abce" candidate_uuid: "60733d46f7c941a48f4144a190933f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94b7f8cb3294fb8872f2b69b95a07be"
I20260925 19:01:18.674328 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36ad6bb838f46c6b983052b8646abce" candidate_uuid: "60733d46f7c941a48f4144a190933f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ce859ccc7643cba09b98e565b9d7b2"
I20260925 19:01:18.674383 10929 raft_consensus.cc:3037] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:01:18.674412 10795 raft_consensus.cc:3037] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:01:18.675127 10795 raft_consensus.cc:2454] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60733d46f7c941a48f4144a190933f42 in term 1.
I20260925 19:01:18.675169 10929 raft_consensus.cc:2454] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60733d46f7c941a48f4144a190933f42 in term 1.
I20260925 19:01:18.675333 10997 leader_election.cc:302] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60733d46f7c941a48f4144a190933f42, c94b7f8cb3294fb8872f2b69b95a07be; no voters:
I20260925 19:01:18.675450 11130 raft_consensus.cc:2781] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:01:18.675604 11130 raft_consensus.cc:686] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 1 LEADER]: Becoming Leader. State: Replica: 60733d46f7c941a48f4144a190933f42, State: Running, Role: LEADER
I20260925 19:01:18.675694 11130 consensus_queue.cc:237] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 } } peers { permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 } } peers { permanent_uuid: "60733d46f7c941a48f4144a190933f42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33637 } }
I20260925 19:01:18.676496 10650 catalog_manager.cc:5495] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60733d46f7c941a48f4144a190933f42 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "60733d46f7c941a48f4144a190933f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60733d46f7c941a48f4144a190933f42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33637 } health_report { overall_health: HEALTHY } } }
I20260925 19:01:18.703128 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.700660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.0.98.3:33637)
0925 19:01:18.700718 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:18.700720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:18.700729 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:01:18.700804 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:18.700805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:18.700820 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:18.700893 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:01:18.701046 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.701519 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.702283 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:18.702497 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:18.702601 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:18.702816 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:01:18.702914 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:01:18.703006 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:18.703025 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 19:01:18.703046 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:01:18.704686 10929 raft_consensus.cc:1261] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2 [term 1 FOLLOWER]: Refusing update from remote peer 60733d46f7c941a48f4144a190933f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:01:18.704723 10795 raft_consensus.cc:1261] T a36ad6bb838f46c6b983052b8646abce P c94b7f8cb3294fb8872f2b69b95a07be [term 1 FOLLOWER]: Refusing update from remote peer 60733d46f7c941a48f4144a190933f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:01:18.705022 11133 consensus_queue.cc:1035] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ce859ccc7643cba09b98e565b9d7b2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:01:18.705166 11130 consensus_queue.cc:1035] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c94b7f8cb3294fb8872f2b69b95a07be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:01:18.707353 11136 mvcc.cc:205] Tried to move back new op lower bound from 7333326351172153344 to 7333326351056408576. Current Snapshot: MvccSnapshot[applied={T|T < 7333326351172153344}]
I20260925 19:01:18.709797 10648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:01:18.709883 10648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:01:18.710570 10648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 226e6a833ccb4cd89ef707233b411fcd: Sending DeleteTablet for 3 replicas of tablet a36ad6bb838f46c6b983052b8646abce
I20260925 19:01:18.710861 10909 tablet_service.cc:1514] Processing DeleteTablet for tablet a36ad6bb838f46c6b983052b8646abce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:01:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48256
I20260925 19:01:18.710911 11043 tablet_service.cc:1514] Processing DeleteTablet for tablet a36ad6bb838f46c6b983052b8646abce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:01:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49980
I20260925 19:01:18.711105 11145 tablet_replica.cc:330] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: stopping tablet replica
I20260925 19:01:18.711115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 10712
I20260925 19:01:18.711105 10775 tablet_service.cc:1514] Processing DeleteTablet for tablet a36ad6bb838f46c6b983052b8646abce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:01:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40574
I20260925 19:01:18.711925 11145 raft_consensus.cc:2229] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:01:18.712077 11145 raft_consensus.cc:2258] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:01:18.712146 11146 tablet_replica.cc:330] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: stopping tablet replica
I20260925 19:01:18.712231 11146 raft_consensus.cc:2229] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:01:18.712334 11146 raft_consensus.cc:2258] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:01:18.712656 11145 ts_tablet_manager.cc:1905] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:01:18.714005 11145 ts_tablet_manager.cc:1918] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:01:18.714080 11145 log.cc:1196] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/a36ad6bb838f46c6b983052b8646abce
I20260925 19:01:18.714190 11146 ts_tablet_manager.cc:1905] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:01:18.714401 11145 ts_tablet_manager.cc:1939] T a36ad6bb838f46c6b983052b8646abce P c4ce859ccc7643cba09b98e565b9d7b2: Deleting consensus metadata
I20260925 19:01:18.714782 10634 catalog_manager.cc:4841] TS c4ce859ccc7643cba09b98e565b9d7b2 (127.0.98.2:35835): tablet a36ad6bb838f46c6b983052b8646abce (table test-table [id=7300be748ee042b982433d4effd7c864]) successfully deleted
I20260925 19:01:18.715891 11146 ts_tablet_manager.cc:1918] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:01:18.715972 11146 log.cc:1196] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/a36ad6bb838f46c6b983052b8646abce
I20260925 19:01:18.716209 11146 ts_tablet_manager.cc:1939] T a36ad6bb838f46c6b983052b8646abce P 60733d46f7c941a48f4144a190933f42: Deleting consensus metadata
I20260925 19:01:18.716683 10636 catalog_manager.cc:4841] TS 60733d46f7c941a48f4144a190933f42 (127.0.98.3:33637): tablet a36ad6bb838f46c6b983052b8646abce (table test-table [id=7300be748ee042b982433d4effd7c864]) successfully deleted
W20260925 19:01:18.717844 10635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38397 (error 108)
I20260925 19:01:18.718056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 10846
I20260925 19:01:18.718654 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:18.718136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38397 (local address 127.0.0.1:40576)
0925 19:01:18.718591 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:18.718632 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440}
W20260925 19:01:18.718986 10635 catalog_manager.cc:4584] TS c94b7f8cb3294fb8872f2b69b95a07be (127.0.98.1:38397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a36ad6bb838f46c6b983052b8646abce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38397: connect: Connection refused (error 111)
I20260925 19:01:18.724303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 10980
I20260925 19:01:18.729044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 10618
2026-09-25T19:01:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1833 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[11150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[11150](info): set up 2 sockets
Sep 25 19:01:18 dist-test-slave-49bz krb5kdc[11150](info): commencing operation
krb5kdc: starting...
W20260925 19:01:19.718272 392 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:01:19 dist-test-slave-49bz krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:01:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 19:01:19.862867 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOuSOt7/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/postgres -l logfile start
2026-09-25 19:01:22.104 UTC [11180] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 19:01:22.104 UTC [11180] LOG: listening on IPv4 address "127.0.98.20", port 55589
2026-09-25 19:01:22.107 UTC [11180] LOG: listening on Unix socket "/tmp/.s.PGSQL.55589"
2026-09-25 19:01:22.127 UTC [11183] LOG: database system was shut down at 2026-09-25 19:01:20 UTC
2026-09-25 19:01:22.131 UTC [11180] LOG: database system is ready to accept connections
I20260925 19:01:23.026211 392 mini_postgres.cc:96] Postgres bound to 55589
2026-09-25 19:01:23.031 UTC [11191] FATAL: database "slave" does not exist
127.0.98.20:55589 - accepting connections
I20260925 19:01:23.032163 392 mini_ranger.cc:161] Starting Ranger...
I20260925 19:01:23.045885 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 19:01:23.468746 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 19:01:23.468811 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-admin
2026-09-25 19:01:23,728 [I] DB FLAVOR :POSTGRES
2026-09-25 19:01:23,728 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:01:23,728 [I] Checking connection..
2026-09-25 19:01:23,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 19:01:23,977 [I] Checking connection passed.
2026-09-25 19:01:23,977 [I] --------- Verifying version history table ---------
2026-09-25 19:01:23,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:01:24,211 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:01:24,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:01:24,460 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:01:24,461 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 19:01:24,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 19:01:24.674 UTC [11255] WARNING: there is no transaction in progress
2026-09-25 19:01:24,680 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 19:01:24,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:01:24,923 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 19:01:24,923 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:01:24,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 19:01:25,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 19:01:25,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-49bz.c.gcp-upstream.internal','N') ;"
2026-09-25 19:01:25,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:01:25,846 [I] Table x_portal_user does not exist in database ranger
2026-09-25 19:01:25,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:01:26,080 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 19:01:26,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:01:26,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:01:26,538 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 19:01:26,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 19:01:27.411 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.426 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.440 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.454 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.469 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.484 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.534 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.541 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.553 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.562 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.572 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.585 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.593 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.599 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.605 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:27.610 UTC [11363] WARNING: there is no transaction in progress
2026-09-25 19:01:28,162 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 19:01:28,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 19:01:28,369 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 19:01:28,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:01:28,616 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 19:01:28,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:01:28,846 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 19:01:28,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:01:29,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:01:29,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-49bz.c.gcp-upstream.internal';"
2026-09-25 19:01:29,538 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 19:01:29,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 19:01:29,770 [I] DB_PATCHES have already been applied
I20260925 19:01:29.775835 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOuSOt7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 19:01:29.775892 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 19:01:29.778745 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55909
Sep 25, 2026 7:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 7:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 7:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 7:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 7:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 19:01:30.832222 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.800s user 0.000s sys 0.008s
Sep 25, 2026 7:01:49 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 7:01:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 7:01:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 7:01:49 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 7:01:50 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 19:01:50.698809 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 19:01:50 dist-test-slave-49bz krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362910, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 19:01:50.773746 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 19:01:50.782402 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 19:01:51.557478 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 19:01:51.557562 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-kms
I20260925 19:01:51.557579 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:55589
2026-09-25 19:01:51,752 [I] DB FLAVOR :POSTGRES
2026-09-25 19:01:51,752 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:01:51,753 [I] Checking connection
2026-09-25 19:01:51,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 19:01:51,987 [I] connection success
2026-09-25 19:01:51,987 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 19:01:51,987 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 19:01:51,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 19:01:52,225 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 19:01:52,226 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:01:52,226 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 19:01:52,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55589/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 19:01:52,462 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 19:01:52.899605 392 mini_ranger_kms.cc:310] Created kms service
I20260925 19:01:53.051397 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 19:01:53.089630 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 19:01:53 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362910, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:01:53.458807 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 19:01:53.458910 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 19:01:53.458931 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 19:01:53.462456 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11543
I20260925 19:01:53.462561 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36077
Sep 25, 2026 7:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 7:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 7:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 7:01:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 19:01:55 dist-test-slave-49bz krb5kdc[11150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790362915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:55.654821 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.881s user 0.012s sys 0.166s
I20260925 19:01:55.654917 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 19:01:55.654990 392 mini_ranger_kms.cc:401] 127.0.98.20:36077/kms/v1/keys
Sep 25 19:01:55 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 19:01:56 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 19:01:58 dist-test-slave-49bz krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:01:58.508397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37557
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37557
--ranger_config_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36077/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:01:58.586474 11594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:58.586673 11594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:58.586692 11594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:58.587757 11594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:01:58.587795 11594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:58.587806 11594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:58.587818 11594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:01:58.587860 11594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:01:58.589809 11594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36077/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37557
--ranger_config_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:58.590023 11594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:58.590320 11594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:58.595784 11599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:58.595769 11600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:58.595870 11602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:58.596167 11594 server_base.cc:1034] running on GCE node
I20260925 19:01:58.596479 11594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:58.596789 11594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:58.597940 11594 hybrid_clock.cc:648] HybridClock initialized: now 1790362918597925 us; error 24 us; skew 500 ppm
Sep 25 19:01:58 dist-test-slave-49bz krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:58.600749 11594 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:01:58.601207 11594 webserver.cc:460] Webserver started at http://127.0.98.62:37403/ using document root <none> and password file <none>
I20260925 19:01:58.601444 11594 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:58.601495 11594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:58.601622 11594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:01:58 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:01:58.656837 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "19c7fcb7dfc9415ebf21dd64244948eb"
format_stamp: "Formatted at 2026-09-25 19:01:58 on dist-test-slave-49bz"
server_key: "4085258a2fcb0eb528ba93fd10a72e44"
server_key_iv: "b8d4068f39d30d4e36fecc8b196eb0d5"
server_key_version: "kuduclusterkey@0"
I20260925 19:01:58.657294 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "19c7fcb7dfc9415ebf21dd64244948eb"
format_stamp: "Formatted at 2026-09-25 19:01:58 on dist-test-slave-49bz"
server_key: "4085258a2fcb0eb528ba93fd10a72e44"
server_key_iv: "b8d4068f39d30d4e36fecc8b196eb0d5"
server_key_version: "kuduclusterkey@0"
I20260925 19:01:58.658802 11594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:01:58.672008 11609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:58.672169 11594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:01:58.672257 11594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "19c7fcb7dfc9415ebf21dd64244948eb"
format_stamp: "Formatted at 2026-09-25 19:01:58 on dist-test-slave-49bz"
server_key: "4085258a2fcb0eb528ba93fd10a72e44"
server_key_iv: "b8d4068f39d30d4e36fecc8b196eb0d5"
server_key_version: "kuduclusterkey@0"
I20260925 19:01:58.672341 11594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:58.685096 11594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:58.687057 11594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:58.687211 11594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:58.692230 11594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37557
I20260925 19:01:58.692246 11661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37557 every 8 connection(s)
I20260925 19:01:58.692588 11594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:01:58.693186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 11594
I20260925 19:01:58.693226 11662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:01:58.693280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:01:58.693462 392 external_mini_cluster.cc:1428] Setting key 6aaf0fa005e1249f0290b9d73a8d046e
I20260925 19:01:58.696522 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb: Bootstrap starting.
I20260925 19:01:58.697541 11662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb: Neither blocks nor log segments found. Creating new log.
I20260925 19:01:58.697827 11662 log.cc:824] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb: Log is configured to *not* fsync() on all Append() calls
I20260925 19:01:58.698540 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb: No bootstrap required, opened a new log
I20260925 19:01:58.700964 11662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c7fcb7dfc9415ebf21dd64244948eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37557 } }
I20260925 19:01:58.701110 11662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:01:58.701186 11662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c7fcb7dfc9415ebf21dd64244948eb, State: Initialized, Role: FOLLOWER
Sep 25 19:01:58 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:01:58.701349 11662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c7fcb7dfc9415ebf21dd64244948eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37557 } }
I20260925 19:01:58.701484 11662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:01:58.701530 11662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:01:58.701587 11662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:01:58.702207 11662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c7fcb7dfc9415ebf21dd64244948eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37557 } }
I20260925 19:01:58.702328 11662 leader_election.cc:302] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19c7fcb7dfc9415ebf21dd64244948eb; no voters:
I20260925 19:01:58.702498 11662 leader_election.cc:288] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:01:58.702546 11667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:01:58.702780 11667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [term 1 LEADER]: Becoming Leader. State: Replica: 19c7fcb7dfc9415ebf21dd64244948eb, State: Running, Role: LEADER
I20260925 19:01:58.702841 11662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:01:58.702970 11667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c7fcb7dfc9415ebf21dd64244948eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37557 } }
I20260925 19:01:58.703323 11667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19c7fcb7dfc9415ebf21dd64244948eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c7fcb7dfc9415ebf21dd64244948eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37557 } } }
I20260925 19:01:58.703449 11667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [sys.catalog]: This master's current role is: LEADER
I20260925 19:01:58.703603 11667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19c7fcb7dfc9415ebf21dd64244948eb. Latest consensus state: current_term: 1 leader_uuid: "19c7fcb7dfc9415ebf21dd64244948eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c7fcb7dfc9415ebf21dd64244948eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37557 } } }
I20260925 19:01:58.703711 11667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb [sys.catalog]: This master's current role is: LEADER
I20260925 19:01:58.703858 11676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:01:58.704362 11676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:01:58.704663 11662 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 19:01:58.705744 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:58.693988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.98.62:37557)
0925 19:01:58.694188 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:58.694198 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:58.694228 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:01:58.694934 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:58.694944 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:58.695009 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:58.695154 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:58.695904 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:58.697296 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:58.698328 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:58.698932 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:58.699043 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:58.701886 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:58.701891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:58.701891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:58.701902 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:58.703980 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:58.704403 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:58.704406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:58.704406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:58.704509 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:58.704769 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:58.704770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:58.704771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:58.705063 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:58.705190 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:58.705433 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 19:01:58.705543 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260925 19:01:58.707419 11676 catalog_manager.cc:1347] Generated new cluster ID: a7a19325efaa4daa844ee00ebe99fd7a
I20260925 19:01:58.707474 11676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:01:58.716948 11676 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:01:58.718279 11676 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:01:58.727252 11676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb: Generated new TSK 0
I20260925 19:01:58.727466 11676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:01:59.497291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37557
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:01:59.614699 11707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:59.614946 11707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:59.614979 11707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:59.616923 11707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:59.616974 11707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:59.617025 11707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:01:59.618916 11707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:59.619127 11707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:01:59.619357 11707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:59.625070 11713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:59.625094 11712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:59.625227 11707 server_base.cc:1034] running on GCE node
W20260925 19:01:59.625337 11715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:59.625530 11707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:59.625774 11707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:59.626966 11707 hybrid_clock.cc:648] HybridClock initialized: now 1790362919626958 us; error 30 us; skew 500 ppm
Sep 25 19:01:59 dist-test-slave-49bz krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:59.629837 11707 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:01:59.630283 11707 webserver.cc:460] Webserver started at http://127.0.98.1:43235/ using document root <none> and password file <none>
I20260925 19:01:59.630499 11707 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:59.630541 11707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:59.630684 11707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:01:59 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:01:59.642640 11707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c2c0c849b7e49debdccf2d6a4b43a42"
format_stamp: "Formatted at 2026-09-25 19:01:59 on dist-test-slave-49bz"
server_key: "0226dc5ec3072606b6c3d31c8fe19e8e"
server_key_iv: "01105bac856946927423ed0f7837766a"
server_key_version: "kuduclusterkey@0"
I20260925 19:01:59.643055 11707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c2c0c849b7e49debdccf2d6a4b43a42"
format_stamp: "Formatted at 2026-09-25 19:01:59 on dist-test-slave-49bz"
server_key: "0226dc5ec3072606b6c3d31c8fe19e8e"
server_key_iv: "01105bac856946927423ed0f7837766a"
server_key_version: "kuduclusterkey@0"
I20260925 19:01:59.644415 11707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:01:59.661442 11724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:59.661612 11707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:01:59.661712 11707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "8c2c0c849b7e49debdccf2d6a4b43a42"
format_stamp: "Formatted at 2026-09-25 19:01:59 on dist-test-slave-49bz"
server_key: "0226dc5ec3072606b6c3d31c8fe19e8e"
server_key_iv: "01105bac856946927423ed0f7837766a"
server_key_version: "kuduclusterkey@0"
I20260925 19:01:59.661805 11707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:59.678371 11707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:59.680904 11707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:59.681085 11707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:59.681349 11707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:59.681715 11707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:59.681790 11707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:59.681849 11707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:59.681895 11707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:59.688539 11707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36695
I20260925 19:01:59.688799 11837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36695 every 8 connection(s)
I20260925 19:01:59.689425 11707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:01:59.694396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 11707
I20260925 19:01:59.694489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:01:59.694653 392 external_mini_cluster.cc:1428] Setting key 280cf674e92d0c2c9ce9f936a5cbb4a4
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 25 19:01:59 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:01:59.716041 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:59.704140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37557 (local address 127.0.98.1:58557)
0925 19:01:59.704788 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:59.704831 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:59.705407 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:59.705889 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:59.705894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:59.706656 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:59.706924 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:59.706932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:59.707915 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:59.707916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:59.709045 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:59.709048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:59.709550 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:59.709552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:59.709700 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:59.710289 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:59.710300 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:59.711877 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:59.714470 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:59.714475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:59.714477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:59.714729 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:59.714935 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:59.714936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:59.714936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:59.715054 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:59.715371 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:59.715377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:59.715553 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 19:01:59.715659 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 19:01:59.715814 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":120,"threads_started":1}
I20260925 19:01:59.716585 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:59.704516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58557 (local address 127.0.98.62:37557)
0925 19:01:59.705563 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:59.705565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:59.705574 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:01:59.705695 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:59.705696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:59.705709 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:59.705765 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:59.707034 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:59.707815 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:59.709134 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:59.709421 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:59.709511 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:59.711980 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:59.711982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:59.711982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:59.711990 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:59.714384 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:59.714803 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:59.714804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:59.714805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:59.714860 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:59.715120 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:59.715121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:59.715121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:59.715261 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:59.715325 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:59.716399 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0925 19:01:59.716482 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1020,"thread_start_us":65,"threads_started":1}
I20260925 19:01:59.717321 11838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37557
I20260925 19:01:59.717424 11838 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:59.717656 11838 heartbeater.cc:511] Master 127.0.98.62:37557 requested a full tablet report, sending...
I20260925 19:01:59.718307 11626 ts_manager.cc:194] Registered new tserver with Master: 8c2c0c849b7e49debdccf2d6a4b43a42 (127.0.98.1:36695)
I20260925 19:01:59.719167 11626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:01:59.725164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37557
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:01:59.857685 11844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:01:59.857991 11844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:01:59.858067 11844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:01:59.859728 11844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:01:59.859830 11844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:01:59.859907 11844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:01:59.862011 11844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:01:59.862471 11844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
Sep 25 19:01:59 dist-test-slave-49bz krb5kdc[11150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790362919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:59.862771 11844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:01:59.869267 11851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:59.869314 11852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:01:59.869453 11854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:01:59.870705 11844 server_base.cc:1034] running on GCE node
I20260925 19:01:59.870899 11844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:01:59.871132 11844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:01:59.872278 11844 hybrid_clock.cc:648] HybridClock initialized: now 1790362919872258 us; error 31 us; skew 500 ppm
Sep 25 19:01:59 dist-test-slave-49bz krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:01:59.875425 11844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:01:59.875900 11844 webserver.cc:460] Webserver started at http://127.0.98.2:41197/ using document root <none> and password file <none>
I20260925 19:01:59.876271 11844 fs_manager.cc:359] Metadata directory not provided
I20260925 19:01:59.876385 11844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:01:59.876543 11844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:01:59 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:01:59.890153 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "0bfe99d0d35448ddbd48720389f89d2f"
format_stamp: "Formatted at 2026-09-25 19:01:59 on dist-test-slave-49bz"
server_key: "2de287509015f83c69a3fcfe2c90be20"
server_key_iv: "7dc8f958a131ea87f63ae1a0e89ec0cd"
server_key_version: "kuduclusterkey@0"
I20260925 19:01:59.890625 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0bfe99d0d35448ddbd48720389f89d2f"
format_stamp: "Formatted at 2026-09-25 19:01:59 on dist-test-slave-49bz"
server_key: "2de287509015f83c69a3fcfe2c90be20"
server_key_iv: "7dc8f958a131ea87f63ae1a0e89ec0cd"
server_key_version: "kuduclusterkey@0"
I20260925 19:01:59.894431 11844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 19:01:59.910291 11861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:59.910527 11844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 19:01:59.910606 11844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "0bfe99d0d35448ddbd48720389f89d2f"
format_stamp: "Formatted at 2026-09-25 19:01:59 on dist-test-slave-49bz"
server_key: "2de287509015f83c69a3fcfe2c90be20"
server_key_iv: "7dc8f958a131ea87f63ae1a0e89ec0cd"
server_key_version: "kuduclusterkey@0"
I20260925 19:01:59.910686 11844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:01:59.943504 11844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:01:59.945614 11844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:01:59.945775 11844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:01:59.946004 11844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:01:59.946380 11844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:01:59.946434 11844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:59.946491 11844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:01:59.946521 11844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:01:59.954224 11844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43353
I20260925 19:01:59.954378 11974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43353 every 8 connection(s)
I20260925 19:01:59.954707 11844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:01:59.955149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 11844
I20260925 19:01:59.955241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:01:59.955395 392 external_mini_cluster.cc:1428] Setting key 07c8ad7aba3fd2164389d6d406ba940a
Sep 25 19:01:59 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 19:01:59.966728 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:59.956120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47963 (local address 127.0.98.62:37557)
0925 19:01:59.956182 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:01:59.956185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:01:59.956329 (+ 144us) server_negotiation.cc:407] Connection header received
0925 19:01:59.957113 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:01:59.957114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:01:59.957126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:01:59.957156 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:01:59.957970 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:59.958808 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:59.960398 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:01:59.960744 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:01:59.960800 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:59.963202 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:01:59.963204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:01:59.963204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:01:59.963212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:01:59.964985 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:59.965492 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:59.965494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:59.965494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:59.965554 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:01:59.965850 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:01:59.965851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:01:59.965852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:01:59.965994 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:01:59.966042 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:01:59.966510 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 19:01:59.966603 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 19:01:59.967260 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:01:59.956054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37557 (local address 127.0.98.2:47963)
0925 19:01:59.956199 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:01:59.956239 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:01:59.956994 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:01:59.957344 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:01:59.957349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:01:59.957598 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:01:59.957870 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:59.957875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:59.958919 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:59.958921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:59.960292 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:01:59.960296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:01:59.960896 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:01:59.960896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:01:59.961038 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:01:59.961590 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:01:59.961598 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:01:59.963085 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:01:59.965080 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:59.965086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:59.965088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:59.965397 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:59.965632 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:01:59.965633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:01:59.965633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:01:59.965764 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:01:59.966136 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:01:59.966143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:01:59.966349 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 19:01:59.966861 (+ 512us) client_negotiation.cc:240] Negotiation successful
0925 19:01:59.967022 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260925 19:01:59.967696 11975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37557
I20260925 19:01:59.967777 11975 heartbeater.cc:464] Registering TS with master...
I20260925 19:01:59.967959 11975 heartbeater.cc:511] Master 127.0.98.62:37557 requested a full tablet report, sending...
I20260925 19:01:59.968309 11626 ts_manager.cc:194] Registered new tserver with Master: 0bfe99d0d35448ddbd48720389f89d2f (127.0.98.2:43353)
I20260925 19:01:59.968772 11626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:01:59.987367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37557
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:00.110306 11981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:00.110628 11981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:00.110716 11981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:00.112347 11981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:00.112439 11981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:00.112514 11981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:02:00.114539 11981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:00.114974 11981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:00.115295 11981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:00.121795 11989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:00.122081 11986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:00.121789 11987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:00.122309 11981 server_base.cc:1034] running on GCE node
I20260925 19:02:00.122505 11981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:00.122726 11981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:00.123871 11981 hybrid_clock.cc:648] HybridClock initialized: now 1790362920123845 us; error 41 us; skew 500 ppm
Sep 25 19:02:00 dist-test-slave-49bz krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:00.127010 11981 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:02:00.127525 11981 webserver.cc:460] Webserver started at http://127.0.98.3:38051/ using document root <none> and password file <none>
I20260925 19:02:00.127770 11981 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:00.127838 11981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:00.127972 11981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:02:00 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:00.141018 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf0bad5f8ae344348abd82bc58ca1f6b"
format_stamp: "Formatted at 2026-09-25 19:02:00 on dist-test-slave-49bz"
server_key: "2749e45b6199a8b6d759ad1f93c81048"
server_key_iv: "1d37752cc93655b022ac6821cfe67550"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:00.141549 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf0bad5f8ae344348abd82bc58ca1f6b"
format_stamp: "Formatted at 2026-09-25 19:02:00 on dist-test-slave-49bz"
server_key: "2749e45b6199a8b6d759ad1f93c81048"
server_key_iv: "1d37752cc93655b022ac6821cfe67550"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:00.150617 11981 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.000s
I20260925 19:02:00.168586 11996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:00.168833 11981 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260925 19:02:00.168919 11981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "bf0bad5f8ae344348abd82bc58ca1f6b"
format_stamp: "Formatted at 2026-09-25 19:02:00 on dist-test-slave-49bz"
server_key: "2749e45b6199a8b6d759ad1f93c81048"
server_key_iv: "1d37752cc93655b022ac6821cfe67550"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:00.169005 11981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:00.196849 11981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:00.199136 11981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:00.199283 11981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:00.199558 11981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:00.199932 11981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:00.199983 11981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:00.200050 11981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:00.200079 11981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:00.208130 11981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33425
I20260925 19:02:00.208192 12109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33425 every 8 connection(s)
I20260925 19:02:00.208540 11981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:02:00.215544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 11981
I20260925 19:02:00.215643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:02:00.215853 392 external_mini_cluster.cc:1428] Setting key 0d63ce714bb3829cfd738735b9e23a62
Sep 25 19:02:00 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:00.246867 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.231234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42923 (local address 127.0.98.62:37557)
0925 19:02:00.231276 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:00.231278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:00.231469 (+ 191us) server_negotiation.cc:407] Connection header received
0925 19:02:00.232131 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:00.232132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:00.232144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:00.232176 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:00.233493 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.234230 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.237355 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.237657 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.237699 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.243654 (+ 5955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:00.243656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:00.243657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:00.243664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:00.245373 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:00.245832 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:00.245833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:00.245834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:00.245888 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:00.246134 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:00.246135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:00.246135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:00.246285 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:00.246327 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:00.246690 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 19:02:00.246776 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:02:00.247622 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.230936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37557 (local address 127.0.98.3:42923)
0925 19:02:00.231351 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:00.231397 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:00.232013 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:00.232813 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:00.232817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:00.233063 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:00.233375 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.233382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.235603 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.235604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.237239 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.237242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.237802 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.237803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.237939 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.241861 (+ 3922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:00.241872 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:00.243539 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:00.245470 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:00.245476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:00.245479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:00.245749 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:00.245953 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:00.245954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:00.245954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:00.246069 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:00.246373 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:00.246379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:00.246545 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 19:02:00.247243 (+ 698us) client_negotiation.cc:240] Negotiation successful
0925 19:02:00.247401 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":87,"threads_started":1}
I20260925 19:02:00.248212 12110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37557
I20260925 19:02:00.248329 12110 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:00.248503 12110 heartbeater.cc:511] Master 127.0.98.62:37557 requested a full tablet report, sending...
I20260925 19:02:00.248874 11626 ts_manager.cc:194] Registered new tserver with Master: bf0bad5f8ae344348abd82bc58ca1f6b (127.0.98.3:33425)
I20260925 19:02:00.249342 11626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42923
I20260925 19:02:00.254052 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:02:00.265684 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.256423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.0.98.62:37557)
0925 19:02:00.256458 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:00.256460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:00.256729 (+ 269us) server_negotiation.cc:407] Connection header received
0925 19:02:00.256812 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:00.256813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:00.256822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:00.256847 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:00.257842 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.258567 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.259852 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.260115 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.260156 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.262388 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:00.262389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:00.262389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:00.262396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:00.263935 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:00.264879 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:00.264880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:00.264881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:00.264932 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:00.265106 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:00.265106 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:00.265107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:00.265213 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:00.265300 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:00.265509 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 19:02:00.265581 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:02:00.268819 11626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:02:00.271054 11626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:02:00.295593 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.276004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43353 (local address 127.0.0.1:54118)
0925 19:02:00.278378 (+ 2374us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:00.278422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:00.278487 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:00.280066 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:00.280067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:00.280068 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:00.280104 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.280105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.286303 (+ 6198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.286304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.289952 (+ 3648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.289955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.295242 (+ 5287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.295243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.295351 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.295369 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 19:02:00.295448 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:02:00.295502 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2344,"thread_start_us":66,"threads_started":1}
I20260925 19:02:00.297710 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.276857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33425 (local address 127.0.0.1:56638)
0925 19:02:00.279282 (+ 2425us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:00.279291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:00.279329 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:00.279988 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:00.279988 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:00.279990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:00.280013 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.280014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.282742 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.282743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.295970 (+ 13227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.295973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.297468 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.297469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.297536 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.297540 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:02:00.297590 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:02:00.297632 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2410,"thread_start_us":63,"threads_started":1}
I20260925 19:02:00.298202 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.277030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.3:33425)
0925 19:02:00.279440 (+ 2410us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:00.279444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:00.279458 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:02:00.279494 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:00.279496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:00.279511 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:00.279582 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:00.281527 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.282649 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.296094 (+ 13445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.296905 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.296981 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.297053 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:00.298069 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0925 19:02:00.298116 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2375,"thread_start_us":89,"threads_started":1}
I20260925 19:02:00.300390 12030 tablet_service.cc:1467] Processing CreateTablet for tablet bdb47d8e5c524622be5f177c3e3c9507 (DEFAULT_TABLE table=test-table [id=68f6446d425245a2a74f34b32ee37d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:00.300817 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb47d8e5c524622be5f177c3e3c9507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:00.299583 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.276517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36695 (local address 127.0.0.1:36224)
0925 19:02:00.278864 (+ 2347us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:00.278874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:00.278910 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:00.279858 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:00.279861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:00.279863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:00.279925 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.279927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.290153 (+ 10226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.290153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.293576 (+ 3423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.293579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.299336 (+ 5757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.299337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.299410 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.299413 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:02:00.299477 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 19:02:00.299516 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2329,"thread_start_us":65,"threads_started":1}
I20260925 19:02:00.305227 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.276692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.0.98.1:36695)
0925 19:02:00.279657 (+ 2965us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:00.279661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:00.279670 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:02:00.279701 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:00.279703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:00.279717 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:00.279780 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:00.280167 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.290071 (+ 9904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.297094 (+ 7023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.299142 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.299218 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.299275 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:00.305105 (+ 5830us) server_negotiation.cc:299] Negotiation successful
0925 19:02:00.305156 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2283,"thread_start_us":46,"threads_started":1}
I20260925 19:02:00.305974 11772 tablet_service.cc:1467] Processing CreateTablet for tablet bdb47d8e5c524622be5f177c3e3c9507 (DEFAULT_TABLE table=test-table [id=68f6446d425245a2a74f34b32ee37d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:00.306304 11772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb47d8e5c524622be5f177c3e3c9507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:00.309594 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.276280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.0.98.2:43353)
0925 19:02:00.278627 (+ 2347us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:00.278633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:00.278642 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:02:00.278683 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:00.278685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:00.278700 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:00.278766 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:00.284296 (+ 5530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.286179 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.293732 (+ 7553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.294595 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.294675 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.295199 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:00.309430 (+ 14231us) server_negotiation.cc:299] Negotiation successful
0925 19:02:00.309510 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2311,"thread_start_us":71,"threads_started":1}
I20260925 19:02:00.310310 11895 tablet_service.cc:1467] Processing CreateTablet for tablet bdb47d8e5c524622be5f177c3e3c9507 (DEFAULT_TABLE table=test-table [id=68f6446d425245a2a74f34b32ee37d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:00.310612 11895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb47d8e5c524622be5f177c3e3c9507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:00.314172 12129 tablet_bootstrap.cc:492] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: Bootstrap starting.
I20260925 19:02:00.315174 12129 tablet_bootstrap.cc:654] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:00.315548 12129 log.cc:824] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:00.315727 12130 tablet_bootstrap.cc:492] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42: Bootstrap starting.
I20260925 19:02:00.316347 12130 tablet_bootstrap.cc:654] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:00.316742 12131 tablet_bootstrap.cc:492] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: Bootstrap starting.
I20260925 19:02:00.317836 12131 tablet_bootstrap.cc:654] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:00.318135 12130 log.cc:824] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:00.318634 12131 log.cc:824] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:00.319345 12129 tablet_bootstrap.cc:492] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: No bootstrap required, opened a new log
I20260925 19:02:00.319458 12129 ts_tablet_manager.cc:1397] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 19:02:00.321434 12129 raft_consensus.cc:348] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } }
I20260925 19:02:00.321682 12129 raft_consensus.cc:374] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:00.321760 12129 raft_consensus.cc:729] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0bad5f8ae344348abd82bc58ca1f6b, State: Initialized, Role: FOLLOWER
I20260925 19:02:00.321977 12129 consensus_queue.cc:260] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } }
I20260925 19:02:00.322337 12129 ts_tablet_manager.cc:1428] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:02:00.322500 12110 heartbeater.cc:503] Master 127.0.98.62:37557 was elected leader, sending a full tablet report...
I20260925 19:02:00.323863 12130 tablet_bootstrap.cc:492] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42: No bootstrap required, opened a new log
I20260925 19:02:00.323968 12130 ts_tablet_manager.cc:1397] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 19:02:00.326213 12130 raft_consensus.cc:348] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } }
I20260925 19:02:00.326455 12130 raft_consensus.cc:374] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:00.326543 12130 raft_consensus.cc:729] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2c0c849b7e49debdccf2d6a4b43a42, State: Initialized, Role: FOLLOWER
I20260925 19:02:00.326711 12130 consensus_queue.cc:260] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } }
I20260925 19:02:00.326761 12131 tablet_bootstrap.cc:492] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: No bootstrap required, opened a new log
I20260925 19:02:00.326828 12131 ts_tablet_manager.cc:1397] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260925 19:02:00.327025 12130 ts_tablet_manager.cc:1428] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:02:00.327168 11838 heartbeater.cc:503] Master 127.0.98.62:37557 was elected leader, sending a full tablet report...
I20260925 19:02:00.328895 12131 raft_consensus.cc:348] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } }
I20260925 19:02:00.329088 12131 raft_consensus.cc:374] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:00.329157 12131 raft_consensus.cc:729] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bfe99d0d35448ddbd48720389f89d2f, State: Initialized, Role: FOLLOWER
I20260925 19:02:00.329313 12131 consensus_queue.cc:260] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } }
I20260925 19:02:00.329597 12131 ts_tablet_manager.cc:1428] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:02:00.329728 11975 heartbeater.cc:503] Master 127.0.98.62:37557 was elected leader, sending a full tablet report...
I20260925 19:02:00.343528 12137 raft_consensus.cc:482] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:02:00.343690 12137 raft_consensus.cc:504] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } }
I20260925 19:02:00.345144 12135 raft_consensus.cc:482] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:02:00.345224 12135 raft_consensus.cc:504] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } }
I20260925 19:02:00.345515 12135 leader_election.cc:288] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bfe99d0d35448ddbd48720389f89d2f (127.0.98.2:43353), 8c2c0c849b7e49debdccf2d6a4b43a42 (127.0.98.1:36695)
I20260925 19:02:00.350925 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.345680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43353 (local address 127.0.98.3:57643)
0925 19:02:00.345737 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:00.345748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:00.345796 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:00.346957 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:00.346958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:00.346959 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:00.347007 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.347009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.348149 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.348150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.349304 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.349306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.350673 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.350674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.350748 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.350751 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:02:00.350809 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:02:00.350853 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 19:02:00.351519 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.346787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57643 (local address 127.0.98.2:43353)
0925 19:02:00.346834 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:00.346835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:00.346841 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:02:00.346858 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:00.346858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:00.346866 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:00.346907 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:00.347123 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.347786 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.350005 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.350574 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.350636 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.350643 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:00.351426 (+ 783us) server_negotiation.cc:299] Negotiation successful
0925 19:02:00.351461 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 19:02:00.351720 11929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb47d8e5c524622be5f177c3e3c9507" candidate_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bfe99d0d35448ddbd48720389f89d2f" is_pre_election: true
I20260925 19:02:00.351816 11929 raft_consensus.cc:2454] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf0bad5f8ae344348abd82bc58ca1f6b in term 0.
I20260925 19:02:00.352010 11999 leader_election.cc:302] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bfe99d0d35448ddbd48720389f89d2f, bf0bad5f8ae344348abd82bc58ca1f6b; no voters:
I20260925 19:02:00.352114 12135 raft_consensus.cc:2781] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:02:00.352145 12135 raft_consensus.cc:482] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:02:00.352164 12135 raft_consensus.cc:3037] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:00.352983 12135 raft_consensus.cc:504] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } }
I20260925 19:02:00.353104 12135 leader_election.cc:288] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [CANDIDATE]: Term 1 election: Requested vote from peers 0bfe99d0d35448ddbd48720389f89d2f (127.0.98.2:43353), 8c2c0c849b7e49debdccf2d6a4b43a42 (127.0.98.1:36695)
I20260925 19:02:00.353511 11929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb47d8e5c524622be5f177c3e3c9507" candidate_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bfe99d0d35448ddbd48720389f89d2f"
I20260925 19:02:00.353577 11929 raft_consensus.cc:3037] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:00.354370 11929 raft_consensus.cc:2454] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf0bad5f8ae344348abd82bc58ca1f6b in term 1.
I20260925 19:02:00.354604 11999 leader_election.cc:302] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bfe99d0d35448ddbd48720389f89d2f, bf0bad5f8ae344348abd82bc58ca1f6b; no voters:
I20260925 19:02:00.354708 12135 raft_consensus.cc:2781] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:02:00.354871 12135 raft_consensus.cc:686] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 1 LEADER]: Becoming Leader. State: Replica: bf0bad5f8ae344348abd82bc58ca1f6b, State: Running, Role: LEADER
I20260925 19:02:00.354959 12135 consensus_queue.cc:237] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } }
I20260925 19:02:00.355669 11626 catalog_manager.cc:5495] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b reported cstate change: term changed from 0 to 1, leader changed from <none> to bf0bad5f8ae344348abd82bc58ca1f6b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 } health_report { overall_health: UNKNOWN } } }
I20260925 19:02:00.356794 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.344195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33425 (local address 127.0.98.2:38413)
0925 19:02:00.344287 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:00.344299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:00.344334 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:00.344554 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:00.344555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:00.344556 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:00.344614 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.344616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.345480 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.345481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.346692 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.346695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.356528 (+ 9833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.356529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.356613 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.356617 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:02:00.356676 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 19:02:00.356728 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 19:02:00.357160 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.344400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38413 (local address 127.0.98.3:33425)
0925 19:02:00.344436 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:00.344438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:00.344445 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:02:00.344466 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:00.344467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:00.344476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:00.344515 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:00.344671 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.345314 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.349393 (+ 4079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.349906 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.349964 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.349973 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:00.357064 (+ 7091us) server_negotiation.cc:299] Negotiation successful
0925 19:02:00.357105 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:02:00.357353 12064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb47d8e5c524622be5f177c3e3c9507" candidate_uuid: "0bfe99d0d35448ddbd48720389f89d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" is_pre_election: true
I20260925 19:02:00.361344 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.345786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36695 (local address 127.0.98.3:52773)
0925 19:02:00.350992 (+ 5206us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:00.351001 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:00.351037 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:00.357876 (+ 6839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:00.357877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:00.357879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:00.357912 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.357913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.358839 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.358840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.359966 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.359969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.361103 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.361104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.361165 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.361182 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 19:02:00.361243 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 19:02:00.361276 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5197,"thread_start_us":86,"threads_started":1}
I20260925 19:02:00.361637 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.345904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52773 (local address 127.0.98.1:36695)
0925 19:02:00.345940 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:00.345942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:00.351147 (+ 5205us) server_negotiation.cc:407] Connection header received
0925 19:02:00.351173 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:00.351174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:00.351186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:00.351229 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:00.358007 (+ 6778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.358759 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.360381 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.360996 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.361063 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.361072 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:00.361533 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 19:02:00.361576 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 19:02:00.362103 11787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb47d8e5c524622be5f177c3e3c9507" candidate_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" is_pre_election: true
I20260925 19:02:00.362263 11787 raft_consensus.cc:2454] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf0bad5f8ae344348abd82bc58ca1f6b in term 0.
I20260925 19:02:00.362380 11786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb47d8e5c524622be5f177c3e3c9507" candidate_uuid: "bf0bad5f8ae344348abd82bc58ca1f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42"
I20260925 19:02:00.362437 11786 raft_consensus.cc:3037] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:00.363230 11786 raft_consensus.cc:2454] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf0bad5f8ae344348abd82bc58ca1f6b in term 1.
I20260925 19:02:00.368250 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.363687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36695 (local address 127.0.98.2:40905)
0925 19:02:00.363824 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:00.363832 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:00.363874 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:00.364005 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:00.364005 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:00.364006 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:00.364035 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.364036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.364753 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.364754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.365733 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:00.365735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.368016 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.368017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:00.368095 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.368097 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:02:00.368138 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 19:02:00.368181 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260925 19:02:00.368603 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.363761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40905 (local address 127.0.98.1:36695)
0925 19:02:00.363800 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:00.363802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:00.363916 (+ 114us) server_negotiation.cc:407] Connection header received
0925 19:02:00.363934 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:00.363935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:00.363942 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:00.363977 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:00.364087 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.364681 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.365836 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.367916 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.367980 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.367990 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:00.368515 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 19:02:00.368550 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:02:00.368821 11786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb47d8e5c524622be5f177c3e3c9507" candidate_uuid: "0bfe99d0d35448ddbd48720389f89d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" is_pre_election: true
I20260925 19:02:00.368914 11786 raft_consensus.cc:2379] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0bfe99d0d35448ddbd48720389f89d2f in current term 1: Already voted for candidate bf0bad5f8ae344348abd82bc58ca1f6b in this term.
I20260925 19:02:00.369134 11862 leader_election.cc:302] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfe99d0d35448ddbd48720389f89d2f; no voters: 8c2c0c849b7e49debdccf2d6a4b43a42, bf0bad5f8ae344348abd82bc58ca1f6b
I20260925 19:02:00.369415 12137 leader_election.cc:288] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf0bad5f8ae344348abd82bc58ca1f6b (127.0.98.3:33425), 8c2c0c849b7e49debdccf2d6a4b43a42 (127.0.98.1:36695)
I20260925 19:02:00.369573 12137 raft_consensus.cc:2726] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 19:02:00.373080 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.370023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.0.98.3:33425)
0925 19:02:00.370056 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:00.370058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:00.370066 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:02:00.370130 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:00.370130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:00.370147 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:00.370223 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:02:00.370420 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.371040 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.372020 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:00.372305 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:00.372373 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:00.372613 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:02:00.372721 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:02:00.372786 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:00.372985 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 19:02:00.373001 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:02:00.384891 11786 raft_consensus.cc:1261] T bdb47d8e5c524622be5f177c3e3c9507 P 8c2c0c849b7e49debdccf2d6a4b43a42 [term 1 FOLLOWER]: Refusing update from remote peer bf0bad5f8ae344348abd82bc58ca1f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:02:00.384923 11929 raft_consensus.cc:1261] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 1 FOLLOWER]: Refusing update from remote peer bf0bad5f8ae344348abd82bc58ca1f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:02:00.385175 12139 consensus_queue.cc:1035] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfe99d0d35448ddbd48720389f89d2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:02:00.385273 12135 consensus_queue.cc:1035] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c2c0c849b7e49debdccf2d6a4b43a42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:02:00.396404 12143 mvcc.cc:205] Tried to move back new op lower bound from 7333326521887297536 to 7333326521774346240. Current Snapshot: MvccSnapshot[applied={T|T < 7333326521887297536}]
I20260925 19:02:00.396512 12144 mvcc.cc:205] Tried to move back new op lower bound from 7333326521887297536 to 7333326521774346240. Current Snapshot: MvccSnapshot[applied={T|T < 7333326521887297536}]
I20260925 19:02:00.396876 12145 mvcc.cc:205] Tried to move back new op lower bound from 7333326521887297536 to 7333326521774346240. Current Snapshot: MvccSnapshot[applied={T|T < 7333326521887297536}]
Sep 25 19:02:00 dist-test-slave-49bz krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:00.407677 11626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:02:00.407804 11626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:02:00.408840 11626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19c7fcb7dfc9415ebf21dd64244948eb: Sending DeleteTablet for 3 replicas of tablet bdb47d8e5c524622be5f177c3e3c9507
I20260925 19:02:00.409111 12030 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb47d8e5c524622be5f177c3e3c9507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56638
I20260925 19:02:00.409313 11895 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb47d8e5c524622be5f177c3e3c9507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54118
I20260925 19:02:00.409494 11772 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb47d8e5c524622be5f177c3e3c9507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36224
I20260925 19:02:00.409986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 11707
I20260925 19:02:00.418392 12154 tablet_replica.cc:330] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: stopping tablet replica
I20260925 19:02:00.421953 12154 raft_consensus.cc:2229] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:02:00.422407 12154 raft_consensus.cc:2258] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:02:00.422698 12153 tablet_replica.cc:330] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: stopping tablet replica
I20260925 19:02:00.422801 12153 raft_consensus.cc:2229] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:02:00.422950 12153 raft_consensus.cc:2258] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:02:00.423404 12154 ts_tablet_manager.cc:1905] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:02:00.423429 12153 ts_tablet_manager.cc:1905] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:02:00.425052 12153 ts_tablet_manager.cc:1918] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:02:00.425191 12153 log.cc:1196] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/bdb47d8e5c524622be5f177c3e3c9507
I20260925 19:02:00.425495 12154 ts_tablet_manager.cc:1918] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:02:00.425540 12153 ts_tablet_manager.cc:1939] T bdb47d8e5c524622be5f177c3e3c9507 P bf0bad5f8ae344348abd82bc58ca1f6b: Deleting consensus metadata
I20260925 19:02:00.425810 12154 log.cc:1196] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/bdb47d8e5c524622be5f177c3e3c9507
I20260925 19:02:00.425926 11611 catalog_manager.cc:4841] TS bf0bad5f8ae344348abd82bc58ca1f6b (127.0.98.3:33425): tablet bdb47d8e5c524622be5f177c3e3c9507 (table test-table [id=68f6446d425245a2a74f34b32ee37d01]) successfully deleted
I20260925 19:02:00.426445 12154 ts_tablet_manager.cc:1939] T bdb47d8e5c524622be5f177c3e3c9507 P 0bfe99d0d35448ddbd48720389f89d2f: Deleting consensus metadata
I20260925 19:02:00.427906 11612 catalog_manager.cc:4841] TS 0bfe99d0d35448ddbd48720389f89d2f (127.0.98.2:43353): tablet bdb47d8e5c524622be5f177c3e3c9507 (table test-table [id=68f6446d425245a2a74f34b32ee37d01]) successfully deleted
W20260925 19:02:00.428289 11610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36695 (error 108)
I20260925 19:02:00.430361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 11844
I20260925 19:02:00.443121 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:00.442997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36695 (local address 127.0.0.1:36236)
0925 19:02:00.443042 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:00.443080 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 19:02:00.443131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 11981
W20260925 19:02:00.443953 11610 catalog_manager.cc:4584] TS 8c2c0c849b7e49debdccf2d6a4b43a42 (127.0.98.1:36695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdb47d8e5c524622be5f177c3e3c9507: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36695: connect: Connection refused (error 111)
I20260925 19:02:00.450196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 11594
I20260925 19:02:00.456501 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 19:02:00.800187 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 19:02:00.800299 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 19:02:01.144246 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 19:02:01.144 UTC [11180] LOG: received smart shutdown request
2026-09-25 19:02:01.147 UTC [11180] LOG: background worker "logical replication launcher" (PID 11189) exited with exit code 1
2026-09-25 19:02:01.147 UTC [11184] LOG: shutting down
2026-09-25 19:02:01.294 UTC [11180] LOG: database system is shut down
2026-09-25T19:02:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42669 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:02:01 dist-test-slave-49bz krb5kdc[12165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:02:01 dist-test-slave-49bz krb5kdc[12165](info): set up 2 sockets
Sep 25 19:02:01 dist-test-slave-49bz krb5kdc[12165](info): commencing operation
krb5kdc: starting...
W20260925 19:02:02.385982 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:02:02 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:02:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 19:02:02.530504 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOuSOt7/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/postgres -l logfile start
2026-09-25 19:02:04.724 UTC [12195] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 19:02:04.724 UTC [12195] LOG: listening on IPv4 address "127.0.98.20", port 49025
2026-09-25 19:02:04.727 UTC [12195] LOG: listening on Unix socket "/tmp/.s.PGSQL.49025"
2026-09-25 19:02:04.756 UTC [12198] LOG: database system was shut down at 2026-09-25 19:02:03 UTC
2026-09-25 19:02:04.759 UTC [12195] LOG: database system is ready to accept connections
I20260925 19:02:05.661845 392 mini_postgres.cc:96] Postgres bound to 49025
2026-09-25 19:02:05.667 UTC [12206] FATAL: database "slave" does not exist
127.0.98.20:49025 - accepting connections
I20260925 19:02:05.667847 392 mini_ranger.cc:161] Starting Ranger...
I20260925 19:02:05.681476 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 19:02:06.077602 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 19:02:06.077682 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-admin
2026-09-25 19:02:06,340 [I] DB FLAVOR :POSTGRES
2026-09-25 19:02:06,341 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:02:06,341 [I] Checking connection..
2026-09-25 19:02:06,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 19:02:06,593 [I] Checking connection passed.
2026-09-25 19:02:06,593 [I] --------- Verifying version history table ---------
2026-09-25 19:02:06,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:02:06,841 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:02:06,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:02:07,087 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 19:02:07,088 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 19:02:07,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 19:02:07.322 UTC [12270] WARNING: there is no transaction in progress
2026-09-25 19:02:07,328 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 19:02:07,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 19:02:07,575 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 19:02:07,575 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:02:07,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 19:02:07,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 19:02:08,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-49bz.c.gcp-upstream.internal','N') ;"
2026-09-25 19:02:08,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:02:08,500 [I] Table x_portal_user does not exist in database ranger
2026-09-25 19:02:08,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:02:08,738 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 19:02:08,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:02:08,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:02:09,206 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 19:02:09,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 19:02:10.079 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.093 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.107 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.120 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.134 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.150 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.200 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.206 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.218 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.227 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.236 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.249 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.258 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.264 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.270 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10.276 UTC [12378] WARNING: there is no transaction in progress
2026-09-25 19:02:10,845 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 19:02:10,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 19:02:11,048 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 19:02:11,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 19:02:11,267 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 19:02:11,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 19:02:11,509 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 19:02:11,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 19:02:11,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 19:02:11,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-49bz.c.gcp-upstream.internal';"
2026-09-25 19:02:12,227 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 19:02:12,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 19:02:12,468 [I] DB_PATCHES have already been applied
I20260925 19:02:12.473650 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOuSOt7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 19:02:12.473706 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 19:02:12.476783 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:46979
Sep 25, 2026 7:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 7:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 7:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 7:02:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 7:02:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 19:02:13.556753 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.889s user 0.002s sys 0.006s
Sep 25, 2026 7:02:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 7:02:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 7:02:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 7:02:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 7:02:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 19:02:33.395965 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 19:02:33 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362953, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 19:02:33.473675 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 19:02:33.482522 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 19:02:34.267467 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 19:02:34.267549 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-kms
I20260925 19:02:34.267566 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:49025
2026-09-25 19:02:34,463 [I] DB FLAVOR :POSTGRES
2026-09-25 19:02:34,464 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 19:02:34,464 [I] Checking connection
2026-09-25 19:02:34,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 19:02:34,692 [I] connection success
2026-09-25 19:02:34,692 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 19:02:34,693 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 19:02:34,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 19:02:34,918 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 19:02:34,918 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 19:02:34,918 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 19:02:34,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49025/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 19:02:35,162 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 19:02:35.565362 392 mini_ranger_kms.cc:310] Created kms service
I20260925 19:02:35.748910 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 19:02:35.790303 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 19:02:35 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362953, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:36.181684 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 19:02:36.181972 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOuSOt7/build/release/bin/postgresql.jar:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOuSOt7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 19:02:36.182004 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 19:02:36.185199 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12558
I20260925 19:02:36.185278 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:46945
Sep 25, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 19:02:38 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790362958, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:38.380548 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.907s user 0.009s sys 0.170s
I20260925 19:02:38.380625 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 19:02:38.380661 392 mini_ranger_kms.cc:401] 127.0.98.20:46945/kms/v1/keys
Sep 25 19:02:38 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362958, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 19:02:39 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362958, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 19:02:41 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:02:41.375559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45369
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45369
--ranger_config_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46945/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:41.456027 12609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:41.456215 12609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:41.456235 12609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:41.457350 12609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:02:41.457389 12609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:41.457403 12609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:41.457415 12609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:02:41.457463 12609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:02:41.459026 12609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:46945/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45369
--ranger_config_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:41.459270 12609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:41.459508 12609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:41.465397 12615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:41.465366 12614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:41.465541 12617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:41.465588 12609 server_base.cc:1034] running on GCE node
I20260925 19:02:41.465831 12609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:41.466132 12609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:41.467295 12609 hybrid_clock.cc:648] HybridClock initialized: now 1790362961467265 us; error 45 us; skew 500 ppm
Sep 25 19:02:41 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:41.476223 12609 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:02:41.476728 12609 webserver.cc:460] Webserver started at http://127.0.98.62:42895/ using document root <none> and password file <none>
I20260925 19:02:41.476953 12609 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:41.477012 12609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:41.477141 12609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:02:41 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:41.531948 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "b6308c4a5f8d4a0a8b9b122931f9474a"
format_stamp: "Formatted at 2026-09-25 19:02:41 on dist-test-slave-49bz"
server_key: "7377e627325d8f548e6c6031ae33b6a3"
server_key_iv: "cfc33dd9219a34ab9f7a9e7e8f3c06ad"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:41.532379 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6308c4a5f8d4a0a8b9b122931f9474a"
format_stamp: "Formatted at 2026-09-25 19:02:41 on dist-test-slave-49bz"
server_key: "7377e627325d8f548e6c6031ae33b6a3"
server_key_iv: "cfc33dd9219a34ab9f7a9e7e8f3c06ad"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:41.533825 12609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:02:41.547823 12624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:41.548040 12609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:02:41.548156 12609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "b6308c4a5f8d4a0a8b9b122931f9474a"
format_stamp: "Formatted at 2026-09-25 19:02:41 on dist-test-slave-49bz"
server_key: "7377e627325d8f548e6c6031ae33b6a3"
server_key_iv: "cfc33dd9219a34ab9f7a9e7e8f3c06ad"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:41.548255 12609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:41.560724 12609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:41.562834 12609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:41.562996 12609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:41.567876 12609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45369
I20260925 19:02:41.567901 12676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45369 every 8 connection(s)
I20260925 19:02:41.568246 12609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:02:41.568912 12677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:41.571105 12677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: Bootstrap starting.
I20260925 19:02:41.571656 12677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:41.571944 12677 log.cc:824] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:41.572556 12677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: No bootstrap required, opened a new log
I20260925 19:02:41.574705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 12609
I20260925 19:02:41.574815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:02:41.574664 12677 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } }
I20260925 19:02:41.574843 12677 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:41.574883 12677 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6308c4a5f8d4a0a8b9b122931f9474a, State: Initialized, Role: FOLLOWER
I20260925 19:02:41.575004 392 external_mini_cluster.cc:1428] Setting key 595dcc0d1877a57ea4464a1b84199c89
I20260925 19:02:41.575074 12677 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } }
I20260925 19:02:41.575163 12677 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:02:41.575196 12677 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:02:41.575250 12677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:41.575999 12677 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } }
I20260925 19:02:41.576175 12677 leader_election.cc:302] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6308c4a5f8d4a0a8b9b122931f9474a; no voters:
I20260925 19:02:41.576416 12677 leader_election.cc:288] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:02:41.576879 12677 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:02:41.578325 12677 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 19:02:41.578363 12683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:02:41.578800 12683 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 1 LEADER]: Becoming Leader. State: Replica: b6308c4a5f8d4a0a8b9b122931f9474a, State: Running, Role: LEADER
I20260925 19:02:41.581175 12683 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } }
I20260925 19:02:41.581996 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6308c4a5f8d4a0a8b9b122931f9474a. Latest consensus state: current_term: 1 leader_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } } }
I20260925 19:02:41.582126 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:41.582697 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } } }
I20260925 19:02:41.582803 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [sys.catalog]: This master's current role is: LEADER
Sep 25 19:02:41 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:41.584602 12698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:02:41.585558 12698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:02:41.588989 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:41.575579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.0.98.62:45369)
0925 19:02:41.575758 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:41.575767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:41.575825 (+ 58us) server_negotiation.cc:407] Connection header received
0925 19:02:41.575977 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:41.575984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:41.576035 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:41.576173 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:41.576998 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:41.578471 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:41.580134 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:41.581972 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:41.582113 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:41.585143 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:41.585148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:41.585149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:41.585162 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:41.587082 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:41.587495 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:41.587498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:41.587499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:41.587590 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:41.587773 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:41.587773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:41.587774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:41.588053 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:41.588189 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:41.588656 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 19:02:41.588784 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260925 19:02:41.589675 12698 catalog_manager.cc:1347] Generated new cluster ID: 911079f4e9de4ec29836f180162f1a63
I20260925 19:02:41.589715 12698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:02:41.599398 12698 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:02:41.600063 12698 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:02:41.604233 12698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: Generated new TSK 0
I20260925 19:02:41.604439 12698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:02:42.323550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45369
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:42.447120 12722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:42.447448 12722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:42.447513 12722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:42.449121 12722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:42.449219 12722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:42.449301 12722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:02:42.451396 12722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:46945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:42.451850 12722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:42.452149 12722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:42.458884 12730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:42.459076 12728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:42.459193 12727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:42.460141 12722 server_base.cc:1034] running on GCE node
I20260925 19:02:42.460371 12722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:42.460690 12722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:42.461849 12722 hybrid_clock.cc:648] HybridClock initialized: now 1790362962461822 us; error 43 us; skew 500 ppm
Sep 25 19:02:42 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:42.465171 12722 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:02:42.465679 12722 webserver.cc:460] Webserver started at http://127.0.98.1:46409/ using document root <none> and password file <none>
I20260925 19:02:42.466001 12722 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:42.466120 12722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:42.466313 12722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:02:42 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:42.481618 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "818a33bcc7324ea38ee362b5804de37e"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "cd85f6b243092b111173f84aafccbdec"
server_key_iv: "83fe86594abc9bb0885cc8dc60a4c6a5"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:42.482100 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "818a33bcc7324ea38ee362b5804de37e"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "cd85f6b243092b111173f84aafccbdec"
server_key_iv: "83fe86594abc9bb0885cc8dc60a4c6a5"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:42.484076 12722 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 19:02:42.497244 12738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:42.497429 12722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:42.497535 12722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "818a33bcc7324ea38ee362b5804de37e"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "cd85f6b243092b111173f84aafccbdec"
server_key_iv: "83fe86594abc9bb0885cc8dc60a4c6a5"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:42.497649 12722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:42.510274 12722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:42.512614 12722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:42.512753 12722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:42.513007 12722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:42.513365 12722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:42.513417 12722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:42.513447 12722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:42.513520 12722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:42.520226 12722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34987
I20260925 19:02:42.520757 12722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:02:42.523877 12852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34987 every 8 connection(s)
I20260925 19:02:42.529553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 12722
I20260925 19:02:42.529651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:02:42.529820 392 external_mini_cluster.cc:1428] Setting key e7afdc986923013b3b59d26085e697c6
Sep 25 19:02:42 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:42.541483 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:42.523382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45369 (local address 127.0.98.1:54761)
0925 19:02:42.524121 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:42.524167 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:42.524821 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:42.530454 (+ 5633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:42.530460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:42.530719 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:42.530995 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:42.531003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:42.532030 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:42.532031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:42.535075 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:42.535077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:42.535597 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:42.535598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:42.535725 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:42.536315 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:42.536326 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:42.537945 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:42.539963 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:42.539970 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:42.539972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:42.540240 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:42.540438 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:42.540439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:42.540439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:42.540573 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:42.540836 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:42.540842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:42.541012 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 19:02:42.541114 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 19:02:42.541259 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":80,"threads_started":1}
I20260925 19:02:42.542035 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:42.523021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54761 (local address 127.0.98.62:45369)
0925 19:02:42.530142 (+ 7121us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:42.530144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:42.530155 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:02:42.530183 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:42.530184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:42.530198 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:42.530310 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:42.531114 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:42.531924 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:42.535171 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:42.535454 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:42.535510 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:42.538052 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:42.538054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:42.538054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:42.538061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:42.539875 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:42.540320 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:42.540321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:42.540321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:42.540379 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:42.540637 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:42.540638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:42.540638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:42.540762 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:42.540813 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:42.541846 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0925 19:02:42.541926 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7087,"thread_start_us":80,"threads_started":1}
I20260925 19:02:42.542739 12853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45369
I20260925 19:02:42.542876 12853 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:42.543092 12853 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
I20260925 19:02:42.543625 12641 ts_manager.cc:194] Registered new tserver with Master: 818a33bcc7324ea38ee362b5804de37e (127.0.98.1:34987)
I20260925 19:02:42.544394 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54761
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:02:42.579615 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45369
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:42.716318 12859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:42.716671 12859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:42.716753 12859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:42.719079 12859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:42.719298 12859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:42.719391 12859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:02:42.722290 12859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:46945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:42.722805 12859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:42.723130 12859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:42.730331 12867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:42.730760 12866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:42.730978 12869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:42.731825 12859 server_base.cc:1034] running on GCE node
I20260925 19:02:42.732117 12859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:42.732395 12859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:42.735663 12859 hybrid_clock.cc:648] HybridClock initialized: now 1790362962735638 us; error 35 us; skew 500 ppm
Sep 25 19:02:42 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:42.738920 12859 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:02:42.739426 12859 webserver.cc:460] Webserver started at http://127.0.98.2:46493/ using document root <none> and password file <none>
I20260925 19:02:42.739677 12859 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:42.739728 12859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:42.739848 12859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:02:42 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:42.752238 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "a3ede20bff0f49c48272d5d1b38d00c9"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "4098da8d7aa539f21b9fdda1876c9587"
server_key_iv: "bd1c1365d21b691daeed5e5fc684430e"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:42.752727 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a3ede20bff0f49c48272d5d1b38d00c9"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "4098da8d7aa539f21b9fdda1876c9587"
server_key_iv: "bd1c1365d21b691daeed5e5fc684430e"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:42.758378 12859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260925 19:02:42.776794 12876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Sep 25 19:02:42 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790362962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:42.784984 12859 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.000s sys 0.001s
I20260925 19:02:42.785207 12859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "a3ede20bff0f49c48272d5d1b38d00c9"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "4098da8d7aa539f21b9fdda1876c9587"
server_key_iv: "bd1c1365d21b691daeed5e5fc684430e"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:42.785431 12859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:42.805508 12859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:42.807986 12859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:42.808182 12859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:42.808457 12859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:42.808871 12859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:42.808961 12859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:42.809031 12859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:42.809085 12859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:42.816268 12859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34031
I20260925 19:02:42.816514 12989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34031 every 8 connection(s)
I20260925 19:02:42.816756 12859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:02:42.819484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 12859
I20260925 19:02:42.819594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:02:42.819756 392 external_mini_cluster.cc:1428] Setting key 6ab2f0a7508f13d831b5f78bad46bfad
Sep 25 19:02:42 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:42.828439 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:42.818404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36137 (local address 127.0.98.62:45369)
0925 19:02:42.818472 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:42.818474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:42.818753 (+ 279us) server_negotiation.cc:407] Connection header received
0925 19:02:42.819302 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:42.819303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:42.819315 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:42.819367 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:42.820088 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:42.820801 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:42.822158 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:42.822468 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:42.822560 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:42.825166 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:42.825168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:42.825168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:42.825176 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:42.826946 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:42.827416 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:42.827417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:42.827418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:42.827473 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:42.827730 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:42.827730 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:42.827731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:42.827869 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:42.827943 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:42.828261 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 19:02:42.828344 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 19:02:42.828573 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:42.818409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45369 (local address 127.0.98.2:36137)
0925 19:02:42.818618 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:42.818658 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:42.819176 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:42.819439 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:42.819443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:42.819711 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:42.819972 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:42.819979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:42.820901 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:42.820902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:42.822038 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:42.822041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:42.822576 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:42.822577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:42.822716 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:42.823304 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:42.823313 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:42.825038 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:42.827024 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:42.827030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:42.827032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:42.827335 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:42.827543 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:42.827544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:42.827544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:42.827660 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:42.827937 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:42.827942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:42.828100 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 19:02:42.828216 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 19:02:42.828357 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260925 19:02:42.829180 12990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45369
I20260925 19:02:42.829293 12990 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:42.829480 12990 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
I20260925 19:02:42.829854 12641 ts_manager.cc:194] Registered new tserver with Master: a3ede20bff0f49c48272d5d1b38d00c9 (127.0.98.2:34031)
I20260925 19:02:42.830219 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:02:42.846580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45369
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:42.960469 12996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:42.960767 12996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:42.960814 12996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:42.962484 12996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:42.962589 12996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:42.962644 12996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:02:42.964720 12996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:46945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:42.965082 12996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:42.965375 12996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:42.971491 13004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:42.971823 12996 server_base.cc:1034] running on GCE node
W20260925 19:02:42.972698 13001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:42.972925 13002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:42.973294 12996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:42.973585 12996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:42.974749 12996 hybrid_clock.cc:648] HybridClock initialized: now 1790362962974726 us; error 35 us; skew 500 ppm
Sep 25 19:02:42 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:42.977911 12996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:02:42.978415 12996 webserver.cc:460] Webserver started at http://127.0.98.3:45447/ using document root <none> and password file <none>
I20260925 19:02:42.978667 12996 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:42.978722 12996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:42.978845 12996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 19:02:42 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:42.991050 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "7ad066cae35982f7393a3b8a11c32746"
server_key_iv: "f776c2ffceedc81038c72e48fd32b5b4"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:42.991506 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "7ad066cae35982f7393a3b8a11c32746"
server_key_iv: "f776c2ffceedc81038c72e48fd32b5b4"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:42.994467 12996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 19:02:43.007251 13011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.007473 12996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:43.007570 12996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "7ad066cae35982f7393a3b8a11c32746"
server_key_iv: "f776c2ffceedc81038c72e48fd32b5b4"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:43.007650 12996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:43.019618 12996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:43.021508 12996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:43.021651 12996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:43.021843 12996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:43.022189 12996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:43.022245 12996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.022306 12996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:43.022337 12996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.027652 12996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45241
I20260925 19:02:43.027730 13124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45241 every 8 connection(s)
I20260925 19:02:43.028047 12996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:02:43.034777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 12996
I20260925 19:02:43.034889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:02:43.035110 392 external_mini_cluster.cc:1428] Setting key 50fa4ce0c973a8dd131011a03be90d6c
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:43.048501 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:43.028777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43025 (local address 127.0.98.62:45369)
0925 19:02:43.028849 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:43.028852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:43.030796 (+ 1944us) server_negotiation.cc:407] Connection header received
0925 19:02:43.031262 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:43.031264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:43.031279 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:43.031359 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:43.031883 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.032697 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.033607 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.033949 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.034032 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:43.045143 (+ 11111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:43.045145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:43.045145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:43.045153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:43.046902 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:43.047394 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:43.047395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:43.047396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:43.047449 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:43.047713 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:43.047714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:43.047714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:43.047842 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:43.047882 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:43.048317 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 19:02:43.048404 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:02:43.049274 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:43.028706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45369 (local address 127.0.98.3:43025)
0925 19:02:43.030674 (+ 1968us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:43.030703 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:43.031163 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:43.031423 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:43.031427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:43.031610 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:43.031791 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:43.031795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.032803 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.032804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:43.033499 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:43.033500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.034051 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.034052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:43.034153 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:43.034755 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:43.034766 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:43.044989 (+ 10223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:43.046997 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:43.047003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:43.047006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:43.047306 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:43.047516 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:43.047516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:43.047516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:43.047642 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:43.047928 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:43.047933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:43.048160 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 19:02:43.048854 (+ 694us) client_negotiation.cc:240] Negotiation successful
0925 19:02:43.049035 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1915,"thread_start_us":81,"threads_started":1}
I20260925 19:02:43.049872 13125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45369
I20260925 19:02:43.049961 13125 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:43.050148 13125 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
I20260925 19:02:43.050554 12641 ts_manager.cc:194] Registered new tserver with Master: fd42ef3fbb31428d8b0f3964bebc0b5a (127.0.98.3:45241)
I20260925 19:02:43.051039 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43025
I20260925 19:02:43.054165 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:02:43.054849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 12722
I20260925 19:02:43.061308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 12859
I20260925 19:02:43.067341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 12996
I20260925 19:02:43.071722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 12609
I20260925 19:02:43.076562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45369
--webserver_interface=127.0.98.62
--webserver_port=42895
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45369
--ranger_config_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46945/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46945/kms with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:43.153723 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:43.153920 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:43.153941 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:43.155011 13128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:02:43.155049 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:43.155081 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:43.155094 13128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:02:43.155107 13128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:02:43.156589 13128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46945/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45369
--ranger_config_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:43.156754 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:43.156922 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:43.161456 13134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:43.161446 13136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:43.161446 13133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:43.161533 13128 server_base.cc:1034] running on GCE node
I20260925 19:02:43.161947 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:43.162195 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:43.163331 13128 hybrid_clock.cc:648] HybridClock initialized: now 1790362963163325 us; error 29 us; skew 500 ppm
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:43.165810 13128 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:02:43.166291 13128 webserver.cc:460] Webserver started at http://127.0.98.62:42895/ using document root <none> and password file <none>
I20260925 19:02:43.166474 13128 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:43.166520 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:43.232060 13128 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:43.232815 13144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.232976 13128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:43.233088 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "b6308c4a5f8d4a0a8b9b122931f9474a"
format_stamp: "Formatted at 2026-09-25 19:02:41 on dist-test-slave-49bz"
server_key: "7377e627325d8f548e6c6031ae33b6a3"
server_key_iv: "cfc33dd9219a34ab9f7a9e7e8f3c06ad"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:43.233455 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:43.246145 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:43.248024 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:43.248132 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:43.252978 13128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45369
I20260925 19:02:43.253007 13196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45369 every 8 connection(s)
I20260925 19:02:43.253374 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:02:43.254997 13197 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:02:43.255817 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: Bootstrap starting.
I20260925 19:02:43.261948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 13128
I20260925 19:02:43.262290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34987
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45369
--webserver_port=46409
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46945/kms with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:02:43.265702 13197 log.cc:824] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:43.267874 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:02:43.268239 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: Bootstrap complete.
I20260925 19:02:43.271552 13197 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } }
I20260925 19:02:43.271863 13197 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6308c4a5f8d4a0a8b9b122931f9474a, State: Initialized, Role: FOLLOWER
I20260925 19:02:43.272056 13197 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } }
I20260925 19:02:43.272135 13197 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:02:43.272198 13197 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:02:43.272244 13197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:02:43.273509 13197 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } }
I20260925 19:02:43.273617 13197 leader_election.cc:302] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6308c4a5f8d4a0a8b9b122931f9474a; no voters:
I20260925 19:02:43.273773 13197 leader_election.cc:288] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:02:43.273837 13201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:02:43.274035 13201 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [term 2 LEADER]: Becoming Leader. State: Replica: b6308c4a5f8d4a0a8b9b122931f9474a, State: Running, Role: LEADER
I20260925 19:02:43.274168 13197 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:02:43.274190 13201 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } }
I20260925 19:02:43.274605 13203 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6308c4a5f8d4a0a8b9b122931f9474a. Latest consensus state: current_term: 2 leader_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } } }
I20260925 19:02:43.274616 13202 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6308c4a5f8d4a0a8b9b122931f9474a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45369 } } }
I20260925 19:02:43.274751 13203 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:43.274770 13202 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:43.274926 13209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:02:43.275429 13209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:02:43.275564 13209 catalog_manager.cc:1259] Loaded cluster ID: 911079f4e9de4ec29836f180162f1a63
I20260925 19:02:43.275627 13209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:02:43.275821 13197 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 19:02:43.278687 13209 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:02:43.279188 13209 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: Loaded TSK: 0
I20260925 19:02:43.279479 13209 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:02:43.373608 13199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:43.373885 13199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:43.373921 13199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:43.375614 13199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:43.375736 13199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:43.375795 13199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:02:43.377827 13199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46409
--tserver_master_addrs=127.0.98.62:45369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:43.378080 13199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:43.378393 13199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:43.385947 13237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:43.385948 13235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:43.386068 13234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:43.387331 13199 server_base.cc:1034] running on GCE node
I20260925 19:02:43.387635 13199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:43.387907 13199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:43.389160 13199 hybrid_clock.cc:648] HybridClock initialized: now 1790362963389129 us; error 46 us; skew 500 ppm
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:43.392371 13199 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:02:43.392884 13199 webserver.cc:460] Webserver started at http://127.0.98.1:46409/ using document root <none> and password file <none>
I20260925 19:02:43.393167 13199 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:43.393249 13199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:43.426607 13199 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.000s sys 0.001s
I20260925 19:02:43.427498 13245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.427736 13199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:02:43.427851 13199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "818a33bcc7324ea38ee362b5804de37e"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "cd85f6b243092b111173f84aafccbdec"
server_key_iv: "83fe86594abc9bb0885cc8dc60a4c6a5"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:43.428342 13199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:43.445000 13199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:43.448352 13199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:43.448523 13199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:43.448777 13199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:43.449129 13199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:43.449203 13199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.449263 13199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:43.449295 13199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.456301 13199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34987
I20260925 19:02:43.456779 13199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:02:43.458505 13358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34987 every 8 connection(s)
I20260925 19:02:43.460201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 13199
I20260925 19:02:43.460474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34031
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45369
--webserver_port=46493
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46945/kms with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:43.484311 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:43.459195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42217 (local address 127.0.98.62:45369)
0925 19:02:43.459394 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:43.459403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:43.466523 (+ 7120us) server_negotiation.cc:407] Connection header received
0925 19:02:43.473320 (+ 6797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:43.473329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:43.473388 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:43.473496 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:43.474433 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.475844 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.477294 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.477880 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.477952 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:43.480496 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:43.480501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:43.480501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:43.480511 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:43.482440 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:43.482898 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:43.482901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:43.482902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:43.482997 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:43.483265 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:43.483266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:43.483266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:43.483556 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:43.483620 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:43.483980 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 19:02:43.484102 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260925 19:02:43.485097 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:43.459151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45369 (local address 127.0.98.1:42217)
0925 19:02:43.466350 (+ 7199us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:43.466400 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:43.467209 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:43.473710 (+ 6501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:43.473714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:43.473955 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:43.474221 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:43.474228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.475971 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.475972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:43.477198 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:43.477201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.478074 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.478075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:43.478204 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:43.478847 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:43.478857 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:43.480368 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:43.482527 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:43.482542 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:43.482544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:43.482809 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:43.483067 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:43.483067 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:43.483068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:43.483185 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:43.483653 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:43.483658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:43.483809 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 19:02:43.484711 (+ 902us) client_negotiation.cc:240] Negotiation successful
0925 19:02:43.484860 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7150,"thread_start_us":87,"threads_started":1}
I20260925 19:02:43.486006 13359 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45369
I20260925 19:02:43.486127 13359 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:43.486340 13359 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
W20260925 19:02:43.487135 13359 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45369 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 19:02:43.487215 13359 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:43.487280 13359 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
I20260925 19:02:43.487604 13359 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:43.487680 13359 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
W20260925 19:02:43.487947 13359 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:02:43.562839 13363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:43.563055 13363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:43.563092 13363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:43.564733 13363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:43.564801 13363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:43.564844 13363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:02:43.566586 13363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46493
--tserver_master_addrs=127.0.98.62:45369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:43.566829 13363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:43.567130 13363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:43.572280 13368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:43.572273 13369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:43.572281 13371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:43.574759 13363 server_base.cc:1034] running on GCE node
I20260925 19:02:43.574998 13363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:43.575296 13363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:43.576459 13363 hybrid_clock.cc:648] HybridClock initialized: now 1790362963576470 us; error 59 us; skew 500 ppm
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:43.578897 13363 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:02:43.579380 13363 webserver.cc:460] Webserver started at http://127.0.98.2:46493/ using document root <none> and password file <none>
I20260925 19:02:43.579619 13363 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:43.579696 13363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:43.592386 13363 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:43.598285 13379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.598524 13363 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260925 19:02:43.598625 13363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "a3ede20bff0f49c48272d5d1b38d00c9"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "4098da8d7aa539f21b9fdda1876c9587"
server_key_iv: "bd1c1365d21b691daeed5e5fc684430e"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:43.599051 13363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:43.617029 13363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:43.619262 13363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:43.619443 13363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:43.619727 13363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:43.620096 13363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:43.620188 13363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.620260 13363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:43.620313 13363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.626653 13363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34031
I20260925 19:02:43.626899 13492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34031 every 8 connection(s)
I20260925 19:02:43.627125 13363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:02:43.636283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 13363
I20260925 19:02:43.636564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45241
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45369
--webserver_port=45447
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46945/kms with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:43.643370 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:43.627791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45369 (local address 127.0.98.2:36293)
0925 19:02:43.633302 (+ 5511us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:43.633344 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:43.633983 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:43.634352 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:43.634356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:43.634600 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:43.634862 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:43.634867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.635858 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.635859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:43.637119 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:43.637122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.637680 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.637681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:43.637805 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:43.638386 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:43.638401 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:43.640034 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:43.641843 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:43.641849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:43.641851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:43.642134 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:43.642383 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:43.642384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:43.642384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:43.642497 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:43.642767 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:43.642772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:43.642927 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 19:02:43.643016 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 19:02:43.643153 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5476,"thread_start_us":83,"threads_started":1}
I20260925 19:02:43.643815 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:43.628064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36293 (local address 127.0.98.62:45369)
0925 19:02:43.628113 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:43.628115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:43.633428 (+ 5313us) server_negotiation.cc:407] Connection header received
0925 19:02:43.634108 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:43.634109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:43.634125 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:43.634156 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:43.634971 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.635750 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.637215 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.637527 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.637574 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:43.640143 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:43.640144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:43.640145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:43.640153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:43.641759 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:43.642263 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:43.642264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:43.642265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:43.642320 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:43.642570 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:43.642571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:43.642571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:43.642690 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:43.642743 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:43.643656 (+ 913us) server_negotiation.cc:299] Negotiation successful
0925 19:02:43.643731 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:02:43.644472 13493 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45369
I20260925 19:02:43.644572 13493 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:43.644743 13493 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
W20260925 19:02:43.645335 13493 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45369 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 19:02:43.645391 13493 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:43.645443 13493 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
I20260925 19:02:43.645747 13493 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:43.645810 13493 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
W20260925 19:02:43.646081 13493 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:02:43.758086 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:43.758404 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:43.758450 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:43.760201 13496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:43.760260 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:43.760311 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:02:43.762573 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45447
--tserver_master_addrs=127.0.98.62:45369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:43.762806 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:43.763131 13496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:43.769106 13502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:43.769765 13503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:43.769878 13496 server_base.cc:1034] running on GCE node
W20260925 19:02:43.769994 13505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:43.770206 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:43.770485 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:43.771682 13496 hybrid_clock.cc:648] HybridClock initialized: now 1790362963771665 us; error 33 us; skew 500 ppm
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:43.774612 13496 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:02:43.775231 13496 webserver.cc:460] Webserver started at http://127.0.98.3:45447/ using document root <none> and password file <none>
I20260925 19:02:43.775523 13496 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:43.775635 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:43.795106 13496 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:43.797753 13513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.800872 13496 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260925 19:02:43.800972 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a"
format_stamp: "Formatted at 2026-09-25 19:02:42 on dist-test-slave-49bz"
server_key: "7ad066cae35982f7393a3b8a11c32746"
server_key_iv: "f776c2ffceedc81038c72e48fd32b5b4"
server_key_version: "kuduclusterkey@0"
I20260925 19:02:43.801404 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:43.825718 13496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:43.828248 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:43.828389 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:43.828709 13496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:43.829128 13496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:43.829187 13496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.829218 13496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:43.829236 13496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:43.836134 13496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45241
I20260925 19:02:43.836825 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:02:43.840834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 13496
I20260925 19:02:43.838845 13626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45241 every 8 connection(s)
Sep 25 19:02:43 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:43.850903 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:43.839794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36797 (local address 127.0.98.62:45369)
0925 19:02:43.839842 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:43.839844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:43.840352 (+ 508us) server_negotiation.cc:407] Connection header received
0925 19:02:43.841058 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:43.841059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:43.841073 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:43.841106 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:43.842160 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.842924 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.844456 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.844737 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.844788 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:43.847578 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:43.847579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:43.847580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:43.847587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:43.849301 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:43.849820 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:43.849821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:43.849822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:43.849892 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:43.850195 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:43.850196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:43.850197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:43.850331 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:43.850372 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:43.850744 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 19:02:43.850817 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:02:43.851608 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:43.839505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45369 (local address 127.0.98.3:36797)
0925 19:02:43.840210 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:43.840257 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:43.840918 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:43.841473 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:43.841477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:43.841734 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:43.842009 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:43.842015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.843056 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.843057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:43.844348 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:43.844351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.844894 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.844895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:43.845055 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:43.845675 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:43.845684 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:43.847467 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:43.849413 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:43.849420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:43.849423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:43.849726 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:43.849964 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:43.849965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:43.849965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:43.850092 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:43.850417 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:43.850423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:43.850589 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 19:02:43.851218 (+ 629us) client_negotiation.cc:240] Negotiation successful
0925 19:02:43.851380 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":88,"threads_started":1}
I20260925 19:02:43.853250 13627 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45369
I20260925 19:02:43.853349 13627 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:43.853529 13627 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
W20260925 19:02:43.854184 13627 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45369 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 19:02:43.854274 13627 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:43.854334 13627 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
I20260925 19:02:43.854663 13627 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:43.854732 13627 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
W20260925 19:02:43.855043 13627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:02:43.855062 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:43.841217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39508 (local address 127.0.98.62:45369)
0925 19:02:43.845338 (+ 4121us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:43.845340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:43.845531 (+ 191us) server_negotiation.cc:407] Connection header received
0925 19:02:43.846522 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:43.846523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:43.846536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:43.846597 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:43.847695 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.848439 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.849526 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:43.849828 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:43.849903 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:43.851362 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:43.851363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:43.851363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:43.851371 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:43.853335 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:43.853809 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:43.853810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:43.853810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:43.853879 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:43.854133 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:43.854134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:43.854135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:43.854279 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:43.854437 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:43.854874 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 19:02:43.854960 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4101,"thread_start_us":77,"threads_started":1}
Sep 25 19:02:44 dist-test-slave-49bz krb5kdc[12165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790362964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:44.488137 13359 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:44.488288 13359 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
I20260925 19:02:44.488695 13161 ts_manager.cc:194] Registered new tserver with Master: 818a33bcc7324ea38ee362b5804de37e (127.0.98.1:34987)
I20260925 19:02:44.489656 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42217
I20260925 19:02:44.646260 13493 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:44.646399 13493 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
I20260925 19:02:44.646621 13161 ts_manager.cc:194] Registered new tserver with Master: a3ede20bff0f49c48272d5d1b38d00c9 (127.0.98.2:34031)
I20260925 19:02:44.647105 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36293
Sep 25 19:02:44 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 19:02:44.855261 13627 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:44.855405 13627 heartbeater.cc:511] Master 127.0.98.62:45369 requested a full tablet report, sending...
I20260925 19:02:44.855756 13161 ts_manager.cc:194] Registered new tserver with Master: fd42ef3fbb31428d8b0f3964bebc0b5a (127.0.98.3:45241)
I20260925 19:02:44.856222 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36797
I20260925 19:02:44.861821 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:02:44.870741 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:44.862834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.0.98.62:45369)
0925 19:02:44.863127 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:44.863129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:44.863302 (+ 173us) server_negotiation.cc:407] Connection header received
0925 19:02:44.863322 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:44.863323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:44.863335 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:44.863387 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:44.864078 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.864841 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.865892 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.866165 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.866219 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:44.867646 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:44.867647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:44.867647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:44.867654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:44.869244 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:44.869694 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:44.869695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:44.869695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:44.869754 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:44.869931 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:44.869931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:44.869932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:44.870047 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:44.870133 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:44.870541 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 19:02:44.870634 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":63,"threads_started":1}
I20260925 19:02:44.873028 13161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:02:44.875465 13161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:02:44.896756 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:44.887925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34987 (local address 127.0.0.1:43752)
0925 19:02:44.888875 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:44.888882 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:44.888911 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:44.889400 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:44.889403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:44.889406 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:44.889474 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:44.889476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.893602 (+ 4126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.893603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:44.894771 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:44.894774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.896438 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.896439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:44.896518 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:44.896532 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 19:02:44.896616 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:02:44.896665 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":62,"threads_started":1}
I20260925 19:02:44.898455 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:44.887235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45241 (local address 127.0.0.1:55484)
0925 19:02:44.888679 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:44.888720 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:44.888784 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:44.889538 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:44.889538 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:44.889540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:44.889564 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:44.889576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.891592 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.891594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:44.897140 (+ 5546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:44.897143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.898183 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.898184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:44.898267 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:44.898271 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:02:44.898323 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:02:44.898370 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1418,"thread_start_us":79,"threads_started":1}
I20260925 19:02:44.899322 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:44.887645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.0.98.3:45241)
0925 19:02:44.888616 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:44.888622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:44.889208 (+ 586us) server_negotiation.cc:407] Connection header received
0925 19:02:44.889241 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:44.889243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:44.889258 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:44.889319 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:44.890448 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.891488 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.897237 (+ 5749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.898000 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.898075 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:44.898148 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:44.899200 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0925 19:02:44.899245 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":57,"threads_started":1}
I20260925 19:02:44.899631 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:44.888153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.0.98.1:34987)
0925 19:02:44.889003 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:44.889007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:44.889018 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:02:44.889051 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:44.889053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:44.889068 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:44.889135 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:44.889641 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.893490 (+ 3849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.895515 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.896256 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.896321 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:44.896397 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:44.899503 (+ 3106us) server_negotiation.cc:299] Negotiation successful
0925 19:02:44.899547 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":51,"threads_started":1}
I20260925 19:02:44.900388 13561 tablet_service.cc:1467] Processing CreateTablet for tablet a69684b2bcb04d17bd445e9319ada4fe (DEFAULT_TABLE table=test-table [id=d887ef2f96214e2d8de00c205895936c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:44.900743 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69684b2bcb04d17bd445e9319ada4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:44.901947 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:44.888284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34031 (local address 127.0.0.1:60878)
0925 19:02:44.892774 (+ 4490us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:44.892783 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:44.892825 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:44.898582 (+ 5757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:44.898583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:44.898585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:44.898624 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:44.898626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.899820 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.899821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:44.900802 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:44.900804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.901730 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.901730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:44.901786 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:44.901790 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:02:44.901839 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:02:44.901875 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4473,"thread_start_us":62,"threads_started":1}
I20260925 19:02:44.902318 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:44.887675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.0.98.2:34031)
0925 19:02:44.898341 (+ 10666us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:44.898347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:44.898357 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:02:44.898393 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:44.898395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:44.898411 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:44.898489 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:44.898715 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.899730 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.900896 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:44.901558 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:44.901627 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:44.901694 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:44.902139 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 19:02:44.902180 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10625,"thread_start_us":48,"threads_started":1}
I20260925 19:02:44.903026 13427 tablet_service.cc:1467] Processing CreateTablet for tablet a69684b2bcb04d17bd445e9319ada4fe (DEFAULT_TABLE table=test-table [id=d887ef2f96214e2d8de00c205895936c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:44.903353 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69684b2bcb04d17bd445e9319ada4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:44.905947 13293 tablet_service.cc:1467] Processing CreateTablet for tablet a69684b2bcb04d17bd445e9319ada4fe (DEFAULT_TABLE table=test-table [id=d887ef2f96214e2d8de00c205895936c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:44.906299 13293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69684b2bcb04d17bd445e9319ada4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:44.910310 13663 tablet_bootstrap.cc:492] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e: Bootstrap starting.
I20260925 19:02:44.910419 13661 tablet_bootstrap.cc:492] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: Bootstrap starting.
I20260925 19:02:44.911018 13663 tablet_bootstrap.cc:654] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:44.911098 13661 tablet_bootstrap.cc:654] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:44.911397 13663 log.cc:824] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:44.911465 13661 log.cc:824] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:44.911979 13662 tablet_bootstrap.cc:492] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9: Bootstrap starting.
I20260925 19:02:44.912185 13663 tablet_bootstrap.cc:492] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e: No bootstrap required, opened a new log
I20260925 19:02:44.912246 13663 ts_tablet_manager.cc:1397] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:02:44.912614 13662 tablet_bootstrap.cc:654] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:44.912952 13662 log.cc:824] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:44.913637 13662 tablet_bootstrap.cc:492] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9: No bootstrap required, opened a new log
I20260925 19:02:44.913695 13662 ts_tablet_manager.cc:1397] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:02:44.914964 13663 raft_consensus.cc:348] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 } } peers { permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 } } peers { permanent_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } }
I20260925 19:02:44.915162 13663 raft_consensus.cc:374] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:44.915199 13663 raft_consensus.cc:729] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818a33bcc7324ea38ee362b5804de37e, State: Initialized, Role: FOLLOWER
I20260925 19:02:44.915354 13663 consensus_queue.cc:260] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 } } peers { permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 } } peers { permanent_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } }
I20260925 19:02:44.915589 13663 ts_tablet_manager.cc:1428] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 19:02:44.915689 13359 heartbeater.cc:503] Master 127.0.98.62:45369 was elected leader, sending a full tablet report...
I20260925 19:02:44.916149 13662 raft_consensus.cc:348] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 } } peers { permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 } } peers { permanent_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } }
I20260925 19:02:44.916277 13662 raft_consensus.cc:374] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:44.916301 13662 raft_consensus.cc:729] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ede20bff0f49c48272d5d1b38d00c9, State: Initialized, Role: FOLLOWER
I20260925 19:02:44.916389 13662 consensus_queue.cc:260] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 } } peers { permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 } } peers { permanent_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } }
I20260925 19:02:44.916577 13662 ts_tablet_manager.cc:1428] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:02:44.916916 13493 heartbeater.cc:503] Master 127.0.98.62:45369 was elected leader, sending a full tablet report...
I20260925 19:02:44.918908 13661 tablet_bootstrap.cc:492] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: No bootstrap required, opened a new log
I20260925 19:02:44.919008 13661 ts_tablet_manager.cc:1397] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260925 19:02:44.920780 13661 raft_consensus.cc:348] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 } } peers { permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 } } peers { permanent_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } }
I20260925 19:02:44.920930 13661 raft_consensus.cc:374] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:44.921003 13661 raft_consensus.cc:729] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd42ef3fbb31428d8b0f3964bebc0b5a, State: Initialized, Role: FOLLOWER
I20260925 19:02:44.921168 13661 consensus_queue.cc:260] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 } } peers { permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 } } peers { permanent_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } }
I20260925 19:02:44.921444 13661 ts_tablet_manager.cc:1428] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:02:44.921500 13627 heartbeater.cc:503] Master 127.0.98.62:45369 was elected leader, sending a full tablet report...
Sep 25 19:02:44 dist-test-slave-49bz krb5kdc[12165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260925 19:02:44.960290 13360 tablet.cc:2369] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:02:45.090961 13628 tablet.cc:2369] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:02:45.134274 13494 tablet.cc:2369] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:02:45.291761 13668 raft_consensus.cc:482] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:02:45.291895 13668 raft_consensus.cc:504] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 } } peers { permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 } } peers { permanent_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } }
I20260925 19:02:45.292344 13668 leader_election.cc:288] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd42ef3fbb31428d8b0f3964bebc0b5a (127.0.98.3:45241), 818a33bcc7324ea38ee362b5804de37e (127.0.98.1:34987)
I20260925 19:02:45.295964 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:45.292763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38871 (local address 127.0.98.3:45241)
0925 19:02:45.292813 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:45.292815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:45.292826 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:02:45.292861 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:45.292862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:45.292874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:45.292963 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:45.293165 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:45.293888 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:45.295192 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:45.295563 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:45.295650 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:45.295661 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:45.295846 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 19:02:45.295894 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:02:45.295997 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:45.292509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45241 (local address 127.0.98.2:38871)
0925 19:02:45.292706 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:45.292717 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:45.292802 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:45.292958 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:45.292959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:45.292961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:45.293070 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:45.293071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:45.293972 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:45.293973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:45.295059 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:45.295061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:45.295651 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:45.295652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:45.295765 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:45.295771 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:02:45.295838 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 19:02:45.295879 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":120,"threads_started":1}
I20260925 19:02:45.296150 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:45.292605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58845 (local address 127.0.98.1:34987)
0925 19:02:45.292677 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:45.292680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:45.292815 (+ 135us) server_negotiation.cc:407] Connection header received
0925 19:02:45.292882 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:45.292883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:45.292899 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:45.292970 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:45.293147 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:45.293625 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:45.295105 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:45.295692 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:45.295774 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:45.295784 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:45.295990 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 19:02:45.296042 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:02:45.296363 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:45.292555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34987 (local address 127.0.98.2:58845)
0925 19:02:45.292749 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:45.292759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:45.292823 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:45.292992 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:45.292993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:45.292995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:45.293066 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:45.293068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:45.293710 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:45.293711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:45.294999 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:45.295002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:45.295848 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:45.295849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:45.295908 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:45.295911 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:02:45.296253 (+ 342us) client_negotiation.cc:240] Negotiation successful
0925 19:02:45.296291 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":35,"thread_start_us":97,"threads_started":1}
I20260925 19:02:45.296382 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69684b2bcb04d17bd445e9319ada4fe" candidate_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" is_pre_election: true
I20260925 19:02:45.296540 13581 raft_consensus.cc:2454] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ede20bff0f49c48272d5d1b38d00c9 in term 0.
I20260925 19:02:45.296698 13313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69684b2bcb04d17bd445e9319ada4fe" candidate_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818a33bcc7324ea38ee362b5804de37e" is_pre_election: true
I20260925 19:02:45.296790 13383 leader_election.cc:302] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3ede20bff0f49c48272d5d1b38d00c9, fd42ef3fbb31428d8b0f3964bebc0b5a; no voters:
I20260925 19:02:45.296846 13313 raft_consensus.cc:2454] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ede20bff0f49c48272d5d1b38d00c9 in term 0.
I20260925 19:02:45.296931 13668 raft_consensus.cc:2781] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:02:45.296996 13668 raft_consensus.cc:482] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:02:45.297017 13668 raft_consensus.cc:3037] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:45.297910 13668 raft_consensus.cc:504] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 } } peers { permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 } } peers { permanent_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } }
I20260925 19:02:45.298051 13668 leader_election.cc:288] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [CANDIDATE]: Term 1 election: Requested vote from peers fd42ef3fbb31428d8b0f3964bebc0b5a (127.0.98.3:45241), 818a33bcc7324ea38ee362b5804de37e (127.0.98.1:34987)
I20260925 19:02:45.298210 13313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69684b2bcb04d17bd445e9319ada4fe" candidate_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818a33bcc7324ea38ee362b5804de37e"
I20260925 19:02:45.298295 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69684b2bcb04d17bd445e9319ada4fe" candidate_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a"
I20260925 19:02:45.298323 13313 raft_consensus.cc:3037] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:45.298372 13581 raft_consensus.cc:3037] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:45.299031 13313 raft_consensus.cc:2454] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ede20bff0f49c48272d5d1b38d00c9 in term 1.
I20260925 19:02:45.299197 13581 raft_consensus.cc:2454] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ede20bff0f49c48272d5d1b38d00c9 in term 1.
I20260925 19:02:45.299208 13380 leader_election.cc:302] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818a33bcc7324ea38ee362b5804de37e, a3ede20bff0f49c48272d5d1b38d00c9; no voters:
I20260925 19:02:45.299327 13668 raft_consensus.cc:2781] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:02:45.299475 13668 raft_consensus.cc:686] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 1 LEADER]: Becoming Leader. State: Replica: a3ede20bff0f49c48272d5d1b38d00c9, State: Running, Role: LEADER
I20260925 19:02:45.299569 13668 consensus_queue.cc:237] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 } } peers { permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 } } peers { permanent_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } }
I20260925 19:02:45.300431 13161 catalog_manager.cc:5495] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3ede20bff0f49c48272d5d1b38d00c9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3ede20bff0f49c48272d5d1b38d00c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } health_report { overall_health: HEALTHY } } }
I20260925 19:02:45.331198 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:45.328818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.0.98.2:34031)
0925 19:02:45.328876 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:45.328879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:45.328887 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:02:45.328968 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:45.328969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:45.328991 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:45.329070 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:02:45.329215 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:45.329661 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:45.330385 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:45.330578 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:45.330675 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:45.330823 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:02:45.330933 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:02:45.331028 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:45.331067 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 19:02:45.331088 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:02:45.332989 13581 raft_consensus.cc:1261] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a [term 1 FOLLOWER]: Refusing update from remote peer a3ede20bff0f49c48272d5d1b38d00c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:02:45.333003 13313 raft_consensus.cc:1261] T a69684b2bcb04d17bd445e9319ada4fe P 818a33bcc7324ea38ee362b5804de37e [term 1 FOLLOWER]: Refusing update from remote peer a3ede20bff0f49c48272d5d1b38d00c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:02:45.333353 13681 consensus_queue.cc:1035] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818a33bcc7324ea38ee362b5804de37e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:02:45.333510 13668 consensus_queue.cc:1035] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd42ef3fbb31428d8b0f3964bebc0b5a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:02:45.336524 13684 mvcc.cc:205] Tried to move back new op lower bound from 7333326706001317888 to 7333326705867612160. Current Snapshot: MvccSnapshot[applied={T|T < 7333326706001317888 or (T in {7333326706001317888})}]
I20260925 19:02:45.336774 13685 mvcc.cc:205] Tried to move back new op lower bound from 7333326706001317888 to 7333326705867612160. Current Snapshot: MvccSnapshot[applied={T|T < 7333326706001317888}]
I20260925 19:02:45.338327 13161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:02:45.338424 13161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:02:45.338802 13686 mvcc.cc:205] Tried to move back new op lower bound from 7333326706001317888 to 7333326705867612160. Current Snapshot: MvccSnapshot[applied={T|T < 7333326706001317888}]
I20260925 19:02:45.339442 13161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6308c4a5f8d4a0a8b9b122931f9474a: Sending DeleteTablet for 3 replicas of tablet a69684b2bcb04d17bd445e9319ada4fe
I20260925 19:02:45.339707 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet a69684b2bcb04d17bd445e9319ada4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60878
I20260925 19:02:45.339789 13293 tablet_service.cc:1514] Processing DeleteTablet for tablet a69684b2bcb04d17bd445e9319ada4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43752
I20260925 19:02:45.340235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 13199
I20260925 19:02:45.340247 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet a69684b2bcb04d17bd445e9319ada4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55484
I20260925 19:02:45.340258 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet a69684b2bcb04d17bd445e9319ada4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55484
I20260925 19:02:45.340934 13696 tablet_replica.cc:330] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: stopping tablet replica
I20260925 19:02:45.341010 13696 raft_consensus.cc:2229] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:02:45.341095 13696 raft_consensus.cc:2258] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:02:45.341521 13696 ts_tablet_manager.cc:1905] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:02:45.343266 13696 ts_tablet_manager.cc:1918] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:02:45.343393 13696 log.cc:1196] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/a69684b2bcb04d17bd445e9319ada4fe
I20260925 19:02:45.343711 13696 ts_tablet_manager.cc:1939] T a69684b2bcb04d17bd445e9319ada4fe P fd42ef3fbb31428d8b0f3964bebc0b5a: Deleting consensus metadata
I20260925 19:02:45.344200 13148 catalog_manager.cc:4841] TS fd42ef3fbb31428d8b0f3964bebc0b5a (127.0.98.3:45241): tablet a69684b2bcb04d17bd445e9319ada4fe (table test-table [id=d887ef2f96214e2d8de00c205895936c]) successfully deleted
W20260925 19:02:45.344506 13148 catalog_manager.cc:4808] TS fd42ef3fbb31428d8b0f3964bebc0b5a (127.0.98.3:45241): delete failed for tablet a69684b2bcb04d17bd445e9319ada4fe because the tablet was not found. No further retry: Not found: Tablet not found: a69684b2bcb04d17bd445e9319ada4fe
W20260925 19:02:45.346230 13145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34987 (error 108)
I20260925 19:02:45.346565 13695 tablet_replica.cc:330] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9: stopping tablet replica
I20260925 19:02:45.346709 13695 raft_consensus.cc:2229] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:02:45.346832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 13363
I20260925 19:02:45.346850 13695 raft_consensus.cc:2258] T a69684b2bcb04d17bd445e9319ada4fe P a3ede20bff0f49c48272d5d1b38d00c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:02:45.350965 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:45.350819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34987 (local address 127.0.0.1:43764)
0925 19:02:45.350881 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:45.350930 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:02:45.351140 13145 catalog_manager.cc:4584] TS 818a33bcc7324ea38ee362b5804de37e (127.0.98.1:34987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a69684b2bcb04d17bd445e9319ada4fe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34987: connect: Connection refused (error 111)
I20260925 19:02:45.352370 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:45.352256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34031 (local address 127.0.0.1:60908)
0925 19:02:45.352310 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:45.352337 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 19:02:45.352396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 13496
I20260925 19:02:45.358093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 13128
I20260925 19:02:45.362851 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 19:02:45.696821 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 19:02:45.696905 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 19:02:46.041024 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 19:02:46.041 UTC [12195] LOG: received smart shutdown request
2026-09-25 19:02:46.044 UTC [12195] LOG: background worker "logical replication launcher" (PID 12204) exited with exit code 1
2026-09-25 19:02:46.045 UTC [12199] LOG: shutting down
2026-09-25 19:02:46.196 UTC [12195] LOG: database system is shut down
2026-09-25T19:02:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44905 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:02:46 dist-test-slave-49bz krb5kdc[13706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:02:46 dist-test-slave-49bz krb5kdc[13706](info): set up 2 sockets
Sep 25 19:02:46 dist-test-slave-49bz krb5kdc[13706](info): commencing operation
krb5kdc: starting...
W20260925 19:02:47.294010 392 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:02:47.411422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32771
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32771,127.0.98.61:42111,127.0.98.60:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:47.490088 13719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:47.490366 13719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:47.490406 13719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:47.491541 13719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:02:47.491609 13719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:47.491633 13719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:47.491663 13719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:02:47.491693 13719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:02:47.493165 13719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32771,127.0.98.61:42111,127.0.98.60:41867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:47.493432 13719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:47.493645 13719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:02:47.498389 13719 server_base.cc:1034] running on GCE node
W20260925 19:02:47.498354 13725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:47.498574 13724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:47.498361 13727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:47.499053 13719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:47.499316 13719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:47.500452 13719 hybrid_clock.cc:648] HybridClock initialized: now 1790362967500433 us; error 36 us; skew 500 ppm
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:47.502820 13719 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:02:47.503228 13719 webserver.cc:460] Webserver started at http://127.0.98.62:39945/ using document root <none> and password file <none>
I20260925 19:02:47.503420 13719 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:47.503456 13719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:47.503536 13719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:02:47.504523 13719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "c911ddc4ca624d50a265f5be91693b06"
format_stamp: "Formatted at 2026-09-25 19:02:47 on dist-test-slave-49bz"
server_key: "6b8a148ca673fe7b4bbd166f08ce94cb"
server_key_iv: "e64a4d4a8544005e3122450abcddba31"
server_key_version: "encryptionkey@0"
I20260925 19:02:47.504819 13719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "c911ddc4ca624d50a265f5be91693b06"
format_stamp: "Formatted at 2026-09-25 19:02:47 on dist-test-slave-49bz"
server_key: "6b8a148ca673fe7b4bbd166f08ce94cb"
server_key_iv: "e64a4d4a8544005e3122450abcddba31"
server_key_version: "encryptionkey@0"
I20260925 19:02:47.506057 13719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:02:47.506798 13734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:47.506996 13719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:47.507066 13719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "c911ddc4ca624d50a265f5be91693b06"
format_stamp: "Formatted at 2026-09-25 19:02:47 on dist-test-slave-49bz"
server_key: "6b8a148ca673fe7b4bbd166f08ce94cb"
server_key_iv: "e64a4d4a8544005e3122450abcddba31"
server_key_version: "encryptionkey@0"
I20260925 19:02:47.507114 13719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:47.524024 13719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:47.525895 13719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:47.526003 13719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:47.529666 13719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32771
I20260925 19:02:47.529736 13786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32771 every 8 connection(s)
I20260925 19:02:47.530089 13719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:02:47.530961 13787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:47.532446 13787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:02:47.535903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 13719
I20260925 19:02:47.536028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:02:47.536221 392 external_mini_cluster.cc:1428] Setting key 41a03ea68c59d45161973c4522e4bee1
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:47.542443 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.533127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32771 (local address 127.0.0.1:36232)
0925 19:02:47.533292 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.533339 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:47.533757 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:47.534020 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:47.534022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:47.534370 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:47.534536 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.534541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.535502 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.535503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.536250 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.536252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.536749 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.536750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.536808 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.537378 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:47.537389 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:47.538892 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:47.541018 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.541024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.541026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.541201 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.541453 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.541453 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.541453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.541524 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.541927 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:47.541931 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:47.541980 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 19:02:47.542064 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:02:47.542198 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260925 19:02:47.542472 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.533289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.0.98.62:32771)
0925 19:02:47.533415 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.533419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.533451 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:02:47.533878 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.533879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.533900 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.533990 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.534614 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.535407 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.536343 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.536656 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.536777 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.538968 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.538977 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.538978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.538985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.540934 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.541279 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.541283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.541283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.541386 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.541583 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.541584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.541584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.541861 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.541946 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.542131 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.542328 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260925 19:02:47.543462 13787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260925 19:02:47.543794 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.543674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42111 (local address 127.0.0.1:54878)
0925 19:02:47.543726 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.543769 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 19:02:47.543951 13737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42111: connect: Connection refused (error 111)
I20260925 19:02:47.544397 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.544302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42111 (local address 127.0.0.1:54882)
0925 19:02:47.544353 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.544380 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 19:02:47.544515 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42111: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 19:02:47.558784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42111
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32771,127.0.98.61:42111,127.0.98.60:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:02:47.593742 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } attempt: 1
I20260925 19:02:47.594244 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.594098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42111 (local address 127.0.0.1:54884)
0925 19:02:47.594159 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.594217 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 19:02:47.594625 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.594529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42111 (local address 127.0.0.1:54892)
0925 19:02:47.594573 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.594599 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:02:47.594712 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42111: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42111: connect: Connection refused (error 111)
W20260925 19:02:47.636818 13794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:47.637146 13794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:47.637184 13794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:47.638525 13794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:02:47.638567 13794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:47.638582 13794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:47.638594 13794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:02:47.638651 13794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:02:47.640138 13794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32771,127.0.98.61:42111,127.0.98.60:41867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:47.640368 13794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:47.640605 13794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:47.645315 13802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:47.645335 13800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:47.645294 13799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:47.645447 13794 server_base.cc:1034] running on GCE node
I20260925 19:02:47.645736 13794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:47.646041 13794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:47.647202 13794 hybrid_clock.cc:648] HybridClock initialized: now 1790362967647190 us; error 67 us; skew 500 ppm
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:47.649668 13794 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 19:02:47.650141 13794 webserver.cc:460] Webserver started at http://127.0.98.61:40097/ using document root <none> and password file <none>
I20260925 19:02:47.650427 13794 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:47.650496 13794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:47.650632 13794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:02:47.651666 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/data/instance:
uuid: "eea72f3b6c664fc0b4030e2d095b8327"
format_stamp: "Formatted at 2026-09-25 19:02:47 on dist-test-slave-49bz"
server_key: "e7aac6a2350b63a8f5ce80d6d9d431f4"
server_key_iv: "ba722da8b8c7589224fb4e03f93baf10"
server_key_version: "encryptionkey@0"
I20260925 19:02:47.652009 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/wal/instance:
uuid: "eea72f3b6c664fc0b4030e2d095b8327"
format_stamp: "Formatted at 2026-09-25 19:02:47 on dist-test-slave-49bz"
server_key: "e7aac6a2350b63a8f5ce80d6d9d431f4"
server_key_iv: "ba722da8b8c7589224fb4e03f93baf10"
server_key_version: "encryptionkey@0"
I20260925 19:02:47.653337 13794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:02:47.654042 13809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:47.654219 13794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:02:47.654333 13794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/wal
uuid: "eea72f3b6c664fc0b4030e2d095b8327"
format_stamp: "Formatted at 2026-09-25 19:02:47 on dist-test-slave-49bz"
server_key: "e7aac6a2350b63a8f5ce80d6d9d431f4"
server_key_iv: "ba722da8b8c7589224fb4e03f93baf10"
server_key_version: "encryptionkey@0"
I20260925 19:02:47.654429 13794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:47.662840 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } attempt: 2
I20260925 19:02:47.663221 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.663100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42111 (local address 127.0.0.1:54902)
0925 19:02:47.663159 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.663193 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 19:02:47.663533 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.663442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42111 (local address 127.0.0.1:54912)
0925 19:02:47.663487 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.663509 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:02:47.663609 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42111: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42111: connect: Connection refused (error 111)
I20260925 19:02:47.675287 13794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:47.677536 13794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:47.677712 13794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:47.681612 13794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42111
I20260925 19:02:47.681655 13861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42111 every 8 connection(s)
I20260925 19:02:47.681996 13794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
I20260925 19:02:47.682859 13862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:47.682997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 13794
I20260925 19:02:47.683101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-1/wal/instance
I20260925 19:02:47.683300 392 external_mini_cluster.cc:1428] Setting key cd80ec881f214982dfe4aafcf3fe1bde
I20260925 19:02:47.685099 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:47.696463 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.685814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.0.98.62:32771)
0925 19:02:47.685857 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.685859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.686069 (+ 210us) server_negotiation.cc:407] Connection header received
0925 19:02:47.686599 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.686600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.686613 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.686668 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.687256 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.687715 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.688657 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.688849 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.688924 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.693422 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.693423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.693424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.693430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.695146 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.695553 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.695554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.695555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.695613 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.695855 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.695856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.695856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.695963 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.696035 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.696297 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.696377 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:02:47.696641 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.685780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32771 (local address 127.0.0.1:36234)
0925 19:02:47.685941 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.685972 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:47.686494 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:47.686743 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:47.686747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:47.686984 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:47.687158 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.687161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.687802 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.687803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.688553 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.688555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.688931 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.688932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.689053 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.689531 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:47.689539 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:47.693312 (+ 3773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:47.695232 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.695238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.695240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.695446 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.695683 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.695684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.695685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.695783 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.696026 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:47.696032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:47.696156 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 19:02:47.696276 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 19:02:47.696420 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260925 19:02:47.697192 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 19:02:47.705497 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.697457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42111 (local address 127.0.0.1:54918)
0925 19:02:47.697669 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.697681 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:47.697738 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:47.697893 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:47.697894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:47.698040 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:47.698096 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.698098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.698950 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.698951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.699762 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.699765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.700163 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.700164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.700241 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.700555 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:47.700558 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:47.702711 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:47.704403 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.704404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.704404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.704560 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.704783 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.704783 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.704784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.704831 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.705079 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:47.705080 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:47.705190 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 19:02:47.705299 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 19:02:47.705403 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260925 19:02:47.705579 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.697528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.0.98.61:42111)
0925 19:02:47.697678 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.697681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.697730 (+ 49us) server_negotiation.cc:407] Connection header received
0925 19:02:47.697797 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.697799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.697811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.697869 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.698192 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.698866 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.699841 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.700090 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.700140 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.702785 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.702792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.702793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.702798 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.704331 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.704636 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.704638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.704639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.704725 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.704884 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.704885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.704885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.705006 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.705121 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.705328 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.705447 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260925 19:02:47.706126 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:02:47.706496 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.706391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48152)
0925 19:02:47.706435 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.706480 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:02:47.706702 13813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
I20260925 19:02:47.707077 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.707003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48166)
0925 19:02:47.707042 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.707064 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 19:02:47.707221 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41867: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 19:02:47.708786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41867
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32771,127.0.98.61:42111,127.0.98.60:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:02:47.754761 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } attempt: 3
I20260925 19:02:47.756404 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } attempt: 1
I20260925 19:02:47.756804 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.756675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48176)
0925 19:02:47.756732 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.756764 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 19:02:47.757177 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.757077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48180)
0925 19:02:47.757127 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.757151 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 19:02:47.757287 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41867: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 19:02:47.762036 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.755228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.0.98.61:42111)
0925 19:02:47.755298 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.755300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.755310 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:02:47.755394 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.755395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.755406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.755477 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.755805 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.756395 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.757421 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.757617 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.757674 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.759513 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.759514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.759515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.759523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.760911 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.761252 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.761253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.761254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.761311 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.761521 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.761522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.761523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.761649 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.761725 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.761875 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.761954 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 19:02:47.762051 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.755170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42111 (local address 127.0.0.1:54920)
0925 19:02:47.755233 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.755267 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:47.755324 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:47.755484 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:47.755485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:47.755658 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:47.755720 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.755722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.756489 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.756491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.757329 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.757332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.757698 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.757699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.757777 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.758194 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:47.758199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:47.759410 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:47.761014 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.761015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.761015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.761181 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.761376 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.761376 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.761377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.761459 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.761718 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:47.761719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:47.761763 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 19:02:47.761851 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 19:02:47.761962 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:02:47.762682 13787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:02:47.762992 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.762896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48194)
0925 19:02:47.762942 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.762966 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 19:02:47.763399 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.763296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48208)
0925 19:02:47.763362 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.763385 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 19:02:47.763518 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41867: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
W20260925 19:02:47.788628 13869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:47.788827 13869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:47.788849 13869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:47.789891 13869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:02:47.789927 13869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:47.789939 13869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:47.789952 13869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:02:47.789963 13869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:02:47.791393 13869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32771,127.0.98.61:42111,127.0.98.60:41867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:47.791566 13869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:47.791729 13869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:02:47.794657 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } attempt: 1
I20260925 19:02:47.795058 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.794917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48218)
0925 19:02:47.794977 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.795019 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 19:02:47.795490 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.795388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48228)
0925 19:02:47.795439 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.795470 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 19:02:47.795610 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41867: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
W20260925 19:02:47.796456 13874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:47.796501 13877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:47.796512 13869 server_base.cc:1034] running on GCE node
W20260925 19:02:47.796470 13875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:47.796890 13869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:47.797199 13869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:47.798312 13869 hybrid_clock.cc:648] HybridClock initialized: now 1790362967798301 us; error 44 us; skew 500 ppm
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:47.800597 13869 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 19:02:47.800923 13869 webserver.cc:460] Webserver started at http://127.0.98.60:34659/ using document root <none> and password file <none>
I20260925 19:02:47.801113 13869 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:47.801154 13869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:47.801232 13869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:02:47.802141 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/data/instance:
uuid: "e2e18f5a06e14f7982f9d21ed0986743"
format_stamp: "Formatted at 2026-09-25 19:02:47 on dist-test-slave-49bz"
server_key: "f162d4c4dbfca5aca7e8035390c28b46"
server_key_iv: "5a765ea083f112db5d473adadb7a87a0"
server_key_version: "encryptionkey@0"
I20260925 19:02:47.802515 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/wal/instance:
uuid: "e2e18f5a06e14f7982f9d21ed0986743"
format_stamp: "Formatted at 2026-09-25 19:02:47 on dist-test-slave-49bz"
server_key: "f162d4c4dbfca5aca7e8035390c28b46"
server_key_iv: "5a765ea083f112db5d473adadb7a87a0"
server_key_version: "encryptionkey@0"
I20260925 19:02:47.803761 13869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:47.804455 13884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:47.804622 13869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:02:47.804700 13869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/wal
uuid: "e2e18f5a06e14f7982f9d21ed0986743"
format_stamp: "Formatted at 2026-09-25 19:02:47 on dist-test-slave-49bz"
server_key: "f162d4c4dbfca5aca7e8035390c28b46"
server_key_iv: "5a765ea083f112db5d473adadb7a87a0"
server_key_version: "encryptionkey@0"
I20260925 19:02:47.804752 13869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:47.823221 13869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:47.825120 13869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:47.825250 13869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:47.825481 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } attempt: 2
I20260925 19:02:47.825918 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.825760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48230)
0925 19:02:47.825820 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.825868 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 19:02:47.826332 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.826194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48242)
0925 19:02:47.826241 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.826302 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:02:47.826447 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41867: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41867: connect: Connection refused (error 111)
I20260925 19:02:47.829118 13869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41867
I20260925 19:02:47.829186 13936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41867 every 8 connection(s)
I20260925 19:02:47.829489 13869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
I20260925 19:02:47.830413 13937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:47.831862 13937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:02:47.833077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 13869
I20260925 19:02:47.833170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790362794046639-392-0/minicluster-data/master-2/wal/instance
I20260925 19:02:47.833364 392 external_mini_cluster.cc:1428] Setting key db48feeef1d68f868dc22979bae8a16c
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 19:02:47.841243 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.832510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.0.98.62:32771)
0925 19:02:47.832580 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.832582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.832734 (+ 152us) server_negotiation.cc:407] Connection header received
0925 19:02:47.833304 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.833305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.833314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.833355 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.834116 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.834727 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.835673 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.835838 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.835887 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.838167 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.838168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.838169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.838177 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.839781 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.840258 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.840259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.840259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.840338 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.840576 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.840577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.840578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.840733 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.840803 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.841049 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.841126 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 19:02:47.841363 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.832461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32771 (local address 127.0.0.1:36244)
0925 19:02:47.832622 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.832662 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:47.833208 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:47.833426 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:47.833430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:47.833767 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:47.834026 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.834032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.834819 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.834820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.835590 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.835593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.835914 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.835915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.836046 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.836561 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:47.836569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:47.838099 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:47.839856 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.839862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.839863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.840180 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.840408 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.840409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.840409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.840508 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.840799 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:47.840804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:47.840909 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 19:02:47.840987 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 19:02:47.841121 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260925 19:02:47.841921 13937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:02:47.842862 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.833737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.0.98.60:41867)
0925 19:02:47.833864 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.833868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.833890 (+ 22us) server_negotiation.cc:407] Connection header received
0925 19:02:47.833950 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.833952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.833966 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.834021 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.834681 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.835505 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.836387 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.836659 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.836728 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.839312 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.839319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.839320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.839327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.841135 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.841520 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.841523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.841523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.841611 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.842297 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.842297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.842298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.842494 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.842666 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.842714 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.842786 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:47.848882 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.842175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42111 (local address 127.0.0.1:54932)
0925 19:02:47.842225 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.842236 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:47.842295 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:47.842458 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:47.842458 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:47.842602 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:47.842652 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.842653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.843433 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.843434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.844139 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.844141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.844525 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.844526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.844594 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.845001 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:47.845006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:47.846399 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:47.847952 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.847953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.847954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.848151 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.848316 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.848316 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.848317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.848361 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.848629 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:47.848629 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:47.848678 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 19:02:47.848744 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 19:02:47.848829 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 19:02:47.848963 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.842302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.0.98.61:42111)
0925 19:02:47.842345 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.842347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.842354 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:02:47.842385 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.842386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.842394 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.842435 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.842748 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.843347 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.844217 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.844442 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.844518 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.846461 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.846462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.846462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.846469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.847870 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.848225 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.848226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.848226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.848275 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.848422 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.848423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.848423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.848562 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.848615 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.848791 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.848864 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:02:47.849313 13937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:02:47.851231 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.843707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.0.98.62:32771)
0925 19:02:47.843747 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.843749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.843795 (+ 46us) server_negotiation.cc:407] Connection header received
0925 19:02:47.843843 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.843843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.843852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.843921 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.844360 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.844903 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.845782 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.846113 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.846185 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.848332 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.848333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.848333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.848338 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.850089 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.850461 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.850463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.850463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.850514 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.850712 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.850713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.850713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.850820 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.851020 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.851057 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.851127 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 19:02:47.856208 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.849540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48250)
0925 19:02:47.849577 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.849587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:47.849624 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:47.849722 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:47.849723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:47.849875 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:47.849908 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.849909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.850550 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.850551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.851002 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.851003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.851466 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.851466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.851511 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.851906 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:47.851911 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:47.853309 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:47.855201 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.855202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.855202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.855358 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.855562 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.855562 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.855562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.855638 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.855849 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:47.855850 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:47.855895 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 19:02:47.855978 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:02:47.856062 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 19:02:47.856261 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.849600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.0.98.60:41867)
0925 19:02:47.849628 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.849630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.849644 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:02:47.849667 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.849668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.849676 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.849717 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.849976 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.850466 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.851094 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.851378 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.851428 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.853408 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.853409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.853409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.853415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.855111 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.855427 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.855428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.855429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.855495 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.855696 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.855696 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.855696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.855782 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.855827 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.856001 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.856101 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":14}
I20260925 19:02:47.857808 13937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743: Bootstrap starting.
I20260925 19:02:47.858942 13937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:47.859212 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.851844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.0.98.61:42111)
0925 19:02:47.851881 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.851900 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.851908 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:02:47.851957 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.851958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.851965 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.852006 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.852554 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.853215 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.853873 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.854063 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.854118 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.856646 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.856647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.856648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.856654 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.858084 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.858520 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.858521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.858521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.858565 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.858726 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.858727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.858727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.858807 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.858925 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.859048 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.859123 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:02:47.859293 13937 log.cc:824] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:47.860103 13937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743: No bootstrap required, opened a new log
I20260925 19:02:47.860829 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:02:47.861799 13937 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } }
I20260925 19:02:47.861933 13937 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:47.861992 13937 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e18f5a06e14f7982f9d21ed0986743, State: Initialized, Role: FOLLOWER
I20260925 19:02:47.862131 13937 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } }
I20260925 19:02:47.862517 13937 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:02:47.862490 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } } }
I20260925 19:02:47.862564 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:02:47.866458 13961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:02:47.866542 13961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:02:47.864830 13869 master_runner.cc:186] Error getting master registration for 127.0.98.62:32771: OK, instance_id { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" instance_seqno: 1790362967528588 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:02:47.869973 13869 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:02:47.870812 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.862480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.0.98.60:41867)
0925 19:02:47.862513 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.862515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.862939 (+ 424us) server_negotiation.cc:407] Connection header received
0925 19:02:47.863023 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.863024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.863033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.863061 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.863729 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.864385 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.865318 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.865596 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.865640 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.867165 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.867166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.867167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.867172 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.869328 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.869843 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.869844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.869844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.869897 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.870157 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.870157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.870158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.870335 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.870432 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.870660 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.870734 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:02:47.870874 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.862367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.0.98.61:42111)
0925 19:02:47.862410 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.862413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.862788 (+ 375us) server_negotiation.cc:407] Connection header received
0925 19:02:47.862857 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.862858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.862868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.862917 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.863598 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.864307 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.865445 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.865728 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.865761 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.867875 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.867876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.867876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.867883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.869643 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.870038 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.870040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.870040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.870100 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.870363 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.870364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.870364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.870468 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.870558 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.870708 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.870782 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:02:47.870942 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } attempt: 2
I20260925 19:02:47.871887 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.862222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.0.98.62:32771)
0925 19:02:47.862290 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.862293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.862631 (+ 338us) server_negotiation.cc:407] Connection header received
0925 19:02:47.862719 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.862720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.862730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.862783 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.863487 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.864068 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.865118 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.865402 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.865441 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.866907 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.866908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.866909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.866915 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.868502 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.869843 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.869844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.869844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.869897 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.870092 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.870093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.870093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.870243 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.870822 (+ 579us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.871743 (+ 921us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.871816 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 19:02:47.877494 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.871264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48262)
0925 19:02:47.871333 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.871342 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:47.871374 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:47.871475 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:47.871476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:47.871607 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:47.871632 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.871633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.872279 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.872280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.872954 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.872956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.873249 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.873250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.873301 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.873647 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:47.873652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:47.874924 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:47.876548 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.876549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.876549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.876783 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.876976 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.876977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.876978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.877040 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.877289 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:47.877290 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:47.877325 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 19:02:47.877377 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:02:47.877439 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 19:02:47.877642 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.871367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.0.98.60:41867)
0925 19:02:47.871395 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.871397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.871403 (+ 6us) server_negotiation.cc:407] Connection header received
0925 19:02:47.871418 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.871418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.871425 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.871468 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.871698 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.872205 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.873030 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.873184 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.873226 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.875023 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.875024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.875024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.875030 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.876439 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.876858 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.876859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.876859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.876921 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.877108 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.877109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.877110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.877227 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.877286 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.877414 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.877481 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 19:02:47.879185 13787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06: Bootstrap starting.
I20260925 19:02:47.879914 13787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:47.880203 13787 log.cc:824] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:47.880954 13787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06: No bootstrap required, opened a new log
I20260925 19:02:47.882862 13787 raft_consensus.cc:348] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } }
I20260925 19:02:47.882997 13787 raft_consensus.cc:374] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:47.883029 13787 raft_consensus.cc:729] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c911ddc4ca624d50a265f5be91693b06, State: Initialized, Role: FOLLOWER
I20260925 19:02:47.883150 13787 consensus_queue.cc:260] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } }
I20260925 19:02:47.883431 13787 sys_catalog.cc:564] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:02:47.883405 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } } }
I20260925 19:02:47.883548 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:02:47.885210 13975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:02:47.885288 13975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:02:47.885885 13719 master_runner.cc:186] Error getting master registration for 127.0.98.61:42111: OK, instance_id { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" instance_seqno: 1790362967680573 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:02:47.888091 13719 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:02:47.917596 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } attempt: 3
Sep 25 19:02:47 dist-test-slave-49bz krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 19:02:47.926831 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.917924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41867 (local address 127.0.0.1:48266)
0925 19:02:47.918026 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:47.918039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:47.918081 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:47.918378 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:47.918379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:47.918589 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:47.918660 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.918663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.919695 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.919696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.920442 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:47.920444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.920937 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.920938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:47.920994 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.921596 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:47.921602 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:47.924071 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:47.925773 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.925774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.925775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.925947 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.926152 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:47.926153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:47.926154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:47.926212 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:47.926503 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:47.926504 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:47.926549 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 19:02:47.926628 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:02:47.926730 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 19:02:47.926831 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:47.918171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.0.98.60:41867)
0925 19:02:47.918212 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:47.918214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:47.918221 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:02:47.918244 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:47.918245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:47.918279 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:47.918733 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:47.918777 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.919589 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.920530 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:47.920801 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:47.920880 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:47.924155 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:47.924157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:47.924157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:47.924164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:47.925667 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.926016 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.926018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.926018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.926079 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:47.926312 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:47.926313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:47.926314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:47.926430 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:47.926500 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:47.926653 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 19:02:47.926730 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:02:47.928574 13862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327: Bootstrap starting.
I20260925 19:02:47.929189 13862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:47.929469 13862 log.cc:824] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:47.930183 13862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327: No bootstrap required, opened a new log
I20260925 19:02:47.931665 13862 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } }
I20260925 19:02:47.931792 13862 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:47.931836 13862 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea72f3b6c664fc0b4030e2d095b8327, State: Initialized, Role: FOLLOWER
I20260925 19:02:47.931982 13862 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } }
I20260925 19:02:47.932293 13862 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:02:47.933782 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } } }
I20260925 19:02:47.933909 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:02:47.934227 13989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:02:47.934365 13989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:02:47.936228 13794 master_runner.cc:263] No leader master found from master eea72f3b6c664fc0b4030e2d095b8327
I20260925 19:02:47.936273 13794 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:02:48.016801 13949 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:02:48.016932 13949 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } }
I20260925 19:02:48.017163 13949 leader_election.cc:288] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c911ddc4ca624d50a265f5be91693b06 (127.0.98.62:32771), eea72f3b6c664fc0b4030e2d095b8327 (127.0.98.61:42111)
I20260925 19:02:48.017460 13761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2e18f5a06e14f7982f9d21ed0986743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c911ddc4ca624d50a265f5be91693b06" is_pre_election: true
I20260925 19:02:48.017629 13761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e18f5a06e14f7982f9d21ed0986743 in term 0.
I20260925 19:02:48.017828 13886 leader_election.cc:302] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c911ddc4ca624d50a265f5be91693b06, e2e18f5a06e14f7982f9d21ed0986743; no voters:
I20260925 19:02:48.017966 13949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:02:48.018015 13949 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:02:48.018044 13949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:48.018982 13949 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } }
I20260925 19:02:48.019110 13949 leader_election.cc:288] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [CANDIDATE]: Term 1 election: Requested vote from peers c911ddc4ca624d50a265f5be91693b06 (127.0.98.62:32771), eea72f3b6c664fc0b4030e2d095b8327 (127.0.98.61:42111)
I20260925 19:02:48.019290 13761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2e18f5a06e14f7982f9d21ed0986743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c911ddc4ca624d50a265f5be91693b06"
I20260925 19:02:48.019363 13761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:48.020190 13761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e18f5a06e14f7982f9d21ed0986743 in term 1.
I20260925 19:02:48.020617 13836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2e18f5a06e14f7982f9d21ed0986743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea72f3b6c664fc0b4030e2d095b8327" is_pre_election: true
I20260925 19:02:48.020777 13836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e18f5a06e14f7982f9d21ed0986743 in term 0.
I20260925 19:02:48.020903 13835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2e18f5a06e14f7982f9d21ed0986743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea72f3b6c664fc0b4030e2d095b8327"
I20260925 19:02:48.020968 13835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:48.021778 13835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e18f5a06e14f7982f9d21ed0986743 in term 1.
I20260925 19:02:48.021878 13886 leader_election.cc:302] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c911ddc4ca624d50a265f5be91693b06, e2e18f5a06e14f7982f9d21ed0986743; no voters:
I20260925 19:02:48.022169 13949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:02:48.022347 13949 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [term 1 LEADER]: Becoming Leader. State: Replica: e2e18f5a06e14f7982f9d21ed0986743, State: Running, Role: LEADER
I20260925 19:02:48.022442 13949 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } }
I20260925 19:02:48.022820 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2e18f5a06e14f7982f9d21ed0986743. Latest consensus state: current_term: 1 leader_uuid: "e2e18f5a06e14f7982f9d21ed0986743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } } }
I20260925 19:02:48.022905 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:48.023126 13993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:02:48.023375 13993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:02:48.026124 13835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [term 1 FOLLOWER]: Refusing update from remote peer e2e18f5a06e14f7982f9d21ed0986743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:02:48.026440 13949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:02:48.026705 13740 tablet.cc:2369] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:02:48.027588 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2e18f5a06e14f7982f9d21ed0986743. Latest consensus state: current_term: 1 leader_uuid: "e2e18f5a06e14f7982f9d21ed0986743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } } }
I20260925 19:02:48.027686 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:02:48.028015 13761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [term 1 FOLLOWER]: Refusing update from remote peer e2e18f5a06e14f7982f9d21ed0986743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:02:48.028258 13949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:02:48.029269 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2e18f5a06e14f7982f9d21ed0986743. Latest consensus state: current_term: 1 leader_uuid: "e2e18f5a06e14f7982f9d21ed0986743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } } }
I20260925 19:02:48.029358 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:02:48.030390 13994 mvcc.cc:205] Tried to move back new op lower bound from 7333326717033009152 to 7333326717020516352. Current Snapshot: MvccSnapshot[applied={T|T < 7333326717033009152}]
I20260925 19:02:48.030670 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2e18f5a06e14f7982f9d21ed0986743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } } }
I20260925 19:02:48.030750 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:48.031091 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2e18f5a06e14f7982f9d21ed0986743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } } }
I20260925 19:02:48.031169 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:02:48.032460 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2e18f5a06e14f7982f9d21ed0986743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } } }
I20260925 19:02:48.032549 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743 [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:48.032549 13997 mvcc.cc:205] Tried to move back new op lower bound from 7333326717033009152 to 7333326717020516352. Current Snapshot: MvccSnapshot[applied={T|T < 7333326717033009152}]
I20260925 19:02:48.032914 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2e18f5a06e14f7982f9d21ed0986743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c911ddc4ca624d50a265f5be91693b06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32771 } } peers { permanent_uuid: "eea72f3b6c664fc0b4030e2d095b8327" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42111 } } peers { permanent_uuid: "e2e18f5a06e14f7982f9d21ed0986743" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41867 } } }
I20260925 19:02:48.033007 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:02:48.033633 13995 mvcc.cc:205] Tried to move back new op lower bound from 7333326717033009152 to 7333326717020516352. Current Snapshot: MvccSnapshot[applied={T|T < 7333326717033009152}]
I20260925 19:02:48.034606 13993 catalog_manager.cc:1347] Generated new cluster ID: 1b6d6821b5124232bbc4e9aff0fe4dab
I20260925 19:02:48.034667 13993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:02:48.048537 13993 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:02:48.049273 13993 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:02:48.064941 13993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2e18f5a06e14f7982f9d21ed0986743: Generated new TSK 0
I20260925 19:02:48.065122 13993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:02:48.885677 13975 catalog_manager.cc:1259] Loaded cluster ID: 1b6d6821b5124232bbc4e9aff0fe4dab
I20260925 19:02:48.885744 13975 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06: loading cluster ID for follower catalog manager: success
I20260925 19:02:48.886567 13975 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06: acquiring CA information for follower catalog manager: success
I20260925 19:02:48.886868 13975 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c911ddc4ca624d50a265f5be91693b06: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:02:48.934809 13989 catalog_manager.cc:1259] Loaded cluster ID: 1b6d6821b5124232bbc4e9aff0fe4dab
I20260925 19:02:48.934880 13989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327: loading cluster ID for follower catalog manager: success
I20260925 19:02:48.935689 13989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327: acquiring CA information for follower catalog manager: success
I20260925 19:02:48.935979 13989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eea72f3b6c664fc0b4030e2d095b8327: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:02:49.123265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 13719
I20260925 19:02:49.128285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 13794
I20260925 19:02:49.132750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 13869
2026-09-25T19:02:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2827 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:02:49 dist-test-slave-49bz krb5kdc[14008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:02:49 dist-test-slave-49bz krb5kdc[14008](info): set up 2 sockets
Sep 25 19:02:49 dist-test-slave-49bz krb5kdc[14008](info): commencing operation
krb5kdc: starting...
W20260925 19:02:50.122141 392 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:02:50 dist-test-slave-49bz krb5kdc[14008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:02:50.233345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38975
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:50.309696 14021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:50.309930 14021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:50.309962 14021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:50.311134 14021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:02:50.311199 14021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:50.311223 14021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:50.311244 14021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:02:50.311273 14021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:02:50.312657 14021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:50.312916 14021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:50.313136 14021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:50.317783 14029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:50.317819 14026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:50.317978 14027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:50.318042 14021 server_base.cc:1034] running on GCE node
I20260925 19:02:50.318408 14021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:50.318703 14021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:50.319850 14021 hybrid_clock.cc:648] HybridClock initialized: now 1790362970319841 us; error 27 us; skew 500 ppm
Sep 25 19:02:50 dist-test-slave-49bz krb5kdc[14008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:50.322125 14021 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:02:50.322578 14021 webserver.cc:460] Webserver started at http://127.0.98.62:38185/ using document root <none> and password file <none>
I20260925 19:02:50.322758 14021 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:50.322796 14021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:50.322880 14021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:02:50.323812 14021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "bc8d60c1a62c490495e5742965df97b7"
format_stamp: "Formatted at 2026-09-25 19:02:50 on dist-test-slave-49bz"
server_key: "af2fef06d2317cce0d2796c26fd514f1"
server_key_iv: "9c7b239748772f41b640f7e835418100"
server_key_version: "encryptionkey@0"
I20260925 19:02:50.324092 14021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc8d60c1a62c490495e5742965df97b7"
format_stamp: "Formatted at 2026-09-25 19:02:50 on dist-test-slave-49bz"
server_key: "af2fef06d2317cce0d2796c26fd514f1"
server_key_iv: "9c7b239748772f41b640f7e835418100"
server_key_version: "encryptionkey@0"
I20260925 19:02:50.325330 14021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:02:50.325979 14036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:50.326169 14021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:02:50.326309 14021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "bc8d60c1a62c490495e5742965df97b7"
format_stamp: "Formatted at 2026-09-25 19:02:50 on dist-test-slave-49bz"
server_key: "af2fef06d2317cce0d2796c26fd514f1"
server_key_iv: "9c7b239748772f41b640f7e835418100"
server_key_version: "encryptionkey@0"
I20260925 19:02:50.326376 14021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:50.350353 14021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:50.352588 14021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:50.352690 14021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:50.356324 14021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38975
I20260925 19:02:50.356382 14088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38975 every 8 connection(s)
I20260925 19:02:50.356683 14021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:02:50.357544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 14021
I20260925 19:02:50.357631 14089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:50.357661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:02:50.357863 392 external_mini_cluster.cc:1428] Setting key 8505c52cf81b56e4270dbce845ff3edb
I20260925 19:02:50.360359 14089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7: Bootstrap starting.
I20260925 19:02:50.360929 14089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:50.361176 14089 log.cc:824] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:50.361805 14089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7: No bootstrap required, opened a new log
I20260925 19:02:50.363627 14089 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d60c1a62c490495e5742965df97b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38975 } }
I20260925 19:02:50.363806 14089 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:50.363838 14089 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8d60c1a62c490495e5742965df97b7, State: Initialized, Role: FOLLOWER
I20260925 19:02:50.363953 14089 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d60c1a62c490495e5742965df97b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38975 } }
I20260925 19:02:50.364030 14089 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:02:50.364061 14089 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:02:50.364105 14089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 19:02:50 dist-test-slave-49bz krb5kdc[14008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:50.364840 14089 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d60c1a62c490495e5742965df97b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38975 } }
I20260925 19:02:50.364929 14089 leader_election.cc:302] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc8d60c1a62c490495e5742965df97b7; no voters:
I20260925 19:02:50.365116 14089 leader_election.cc:288] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:02:50.365145 14094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:02:50.365296 14094 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [term 1 LEADER]: Becoming Leader. State: Replica: bc8d60c1a62c490495e5742965df97b7, State: Running, Role: LEADER
I20260925 19:02:50.365423 14089 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:02:50.365434 14094 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d60c1a62c490495e5742965df97b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38975 } }
I20260925 19:02:50.365831 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc8d60c1a62c490495e5742965df97b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d60c1a62c490495e5742965df97b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38975 } } }
I20260925 19:02:50.365962 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc8d60c1a62c490495e5742965df97b7. Latest consensus state: current_term: 1 leader_uuid: "bc8d60c1a62c490495e5742965df97b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d60c1a62c490495e5742965df97b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38975 } } }
I20260925 19:02:50.366848 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [sys.catalog]: This master's current role is: LEADER
W20260925 19:02:50.367341 14109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:02:50.367383 14109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:02:50.366964 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7 [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:50.368285 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:50.358398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.0.98.62:38975)
0925 19:02:50.358580 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:50.358588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:50.358613 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:02:50.359055 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:50.359061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:50.359103 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:50.359268 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:50.359881 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:50.360816 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:50.361665 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:50.362116 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:50.362315 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:50.364821 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:50.364824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:50.364825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:50.364834 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:50.366712 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:50.367112 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:50.367115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:50.367115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:50.367239 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:50.367481 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:50.367483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:50.367483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:50.367701 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:50.367799 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:50.367959 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 19:02:50.368053 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260925 19:02:50.368599 14111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:02:50.368762 14111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:02:50.369207 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:02:50.370223 14111 catalog_manager.cc:1347] Generated new cluster ID: 470d2c7a399e4ae0a5b993a9342c0922
I20260925 19:02:50.370311 14111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:02:50.379487 14111 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:02:50.380169 14111 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:02:50.389719 14111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc8d60c1a62c490495e5742965df97b7: Generated new TSK 0
I20260925 19:02:50.389889 14111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:02:50.423332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 14021
2026-09-25T19:02:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1289 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:02:50 dist-test-slave-49bz krb5kdc[14116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:02:50 dist-test-slave-49bz krb5kdc[14116](info): set up 2 sockets
Sep 25 19:02:50 dist-test-slave-49bz krb5kdc[14116](info): commencing operation
krb5kdc: starting...
W20260925 19:02:51.418366 392 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:02:51 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:02:51.528923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:51.608122 14129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:51.608345 14129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:51.608382 14129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:51.609449 14129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:02:51.609510 14129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:51.609534 14129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:51.609562 14129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:02:51.609584 14129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:02:51.611003 14129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:51.611269 14129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:51.611482 14129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:51.616147 14137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:51.616169 14134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:51.616286 14129 server_base.cc:1034] running on GCE node
W20260925 19:02:51.616169 14135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:51.616592 14129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:51.616856 14129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:51.618022 14129 hybrid_clock.cc:648] HybridClock initialized: now 1790362971618004 us; error 31 us; skew 500 ppm
Sep 25 19:02:51 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:51.620461 14129 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:02:51.620857 14129 webserver.cc:460] Webserver started at http://127.0.98.62:43935/ using document root <none> and password file <none>
I20260925 19:02:51.621080 14129 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:51.621145 14129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:51.621289 14129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:02:51.622301 14129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "34031c0838174e07ac6f48a64d042ce2"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "75bc3acce9d83fdab21bfbf4322accea"
server_key_iv: "ee71fc90c9679513ebbac245a479906f"
server_key_version: "encryptionkey@0"
I20260925 19:02:51.622663 14129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "34031c0838174e07ac6f48a64d042ce2"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "75bc3acce9d83fdab21bfbf4322accea"
server_key_iv: "ee71fc90c9679513ebbac245a479906f"
server_key_version: "encryptionkey@0"
I20260925 19:02:51.623857 14129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:02:51.624491 14144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:51.624636 14129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:02:51.624744 14129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "34031c0838174e07ac6f48a64d042ce2"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "75bc3acce9d83fdab21bfbf4322accea"
server_key_iv: "ee71fc90c9679513ebbac245a479906f"
server_key_version: "encryptionkey@0"
I20260925 19:02:51.624827 14129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:51.646795 14129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:51.649024 14129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:51.649175 14129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:51.652896 14129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36379
I20260925 19:02:51.652972 14196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36379 every 8 connection(s)
I20260925 19:02:51.653301 14129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:02:51.654271 14197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:51.656584 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Bootstrap starting.
I20260925 19:02:51.657133 14197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:51.657411 14197 log.cc:824] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:51.658057 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: No bootstrap required, opened a new log
I20260925 19:02:51.659612 14197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:51.659747 14197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:51.659773 14197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34031c0838174e07ac6f48a64d042ce2, State: Initialized, Role: FOLLOWER
I20260925 19:02:51.659947 14197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:51.660032 14197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:02:51.660080 14197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:02:51.660130 14197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:51.660892 14197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:51.660998 14197 leader_election.cc:302] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34031c0838174e07ac6f48a64d042ce2; no voters:
I20260925 19:02:51.661170 14197 leader_election.cc:288] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:02:51.661201 14200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:02:51.661361 14200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 1 LEADER]: Becoming Leader. State: Replica: 34031c0838174e07ac6f48a64d042ce2, State: Running, Role: LEADER
I20260925 19:02:51.661418 14197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:02:51.661487 14200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:51.661868 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34031c0838174e07ac6f48a64d042ce2. Latest consensus state: current_term: 1 leader_uuid: "34031c0838174e07ac6f48a64d042ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } }
I20260925 19:02:51.662030 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:51.661864 14201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34031c0838174e07ac6f48a64d042ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } }
I20260925 19:02:51.662133 14201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:51.662539 14209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:02:51.663053 14209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:02:51.663182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 14129
I20260925 19:02:51.663277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:02:51.663470 392 external_mini_cluster.cc:1428] Setting key 5f9610e6c3f215f09831d1de1800e6c0
I20260925 19:02:51.664646 14209 catalog_manager.cc:1347] Generated new cluster ID: 2274be48efeb41f8822a473c0dc044fd
I20260925 19:02:51.664697 14209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:02:51 dist-test-slave-49bz krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:51.676994 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:51.663893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.0.98.62:36379)
0925 19:02:51.666632 (+ 2739us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:51.666641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:51.666671 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:02:51.666790 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:51.666797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:51.666853 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:51.666987 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:51.667695 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:51.669086 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:51.670069 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:51.670733 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:51.670853 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:51.673513 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:51.673517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:51.673517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:51.673530 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:51.675407 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:51.675806 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:51.675809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:51.675809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:51.675904 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:51.676101 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:51.676102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:51.676102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:51.676374 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:51.676488 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:51.676651 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 19:02:51.676782 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2689,"thread_start_us":64,"threads_started":1}
I20260925 19:02:51.688832 14209 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:02:51.689376 14209 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:02:51.693563 14209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Generated new TSK 0
I20260925 19:02:51.693727 14209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:02:51.718883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:51.797289 14223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:51.797533 14223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:51.797571 14223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:51.798961 14223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:51.799024 14223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:51.799067 14223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:02:51.800385 14223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:51.800622 14223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:51.800854 14223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:51.805685 14231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:51.805696 14229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:51.805712 14228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:51.805897 14223 server_base.cc:1034] running on GCE node
I20260925 19:02:51.806139 14223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:51.806412 14223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:51.807585 14223 hybrid_clock.cc:648] HybridClock initialized: now 1790362971807558 us; error 39 us; skew 500 ppm
Sep 25 19:02:51 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:51.810081 14223 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:02:51.810555 14223 webserver.cc:460] Webserver started at http://127.0.98.1:45147/ using document root <none> and password file <none>
I20260925 19:02:51.810724 14223 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:51.810770 14223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:51.810850 14223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:02:51.811815 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "bb812bd1639640c0b625ba6b716c537a"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "eb8e359ceef34fc508dee395b00601c8"
server_key_iv: "b39a31b7b4184253fa0643c42cf721c3"
server_key_version: "encryptionkey@0"
I20260925 19:02:51.812103 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bb812bd1639640c0b625ba6b716c537a"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "eb8e359ceef34fc508dee395b00601c8"
server_key_iv: "b39a31b7b4184253fa0643c42cf721c3"
server_key_version: "encryptionkey@0"
I20260925 19:02:51.813279 14223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:02:51.813899 14238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:51.814121 14223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:02:51.814215 14223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "bb812bd1639640c0b625ba6b716c537a"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "eb8e359ceef34fc508dee395b00601c8"
server_key_iv: "b39a31b7b4184253fa0643c42cf721c3"
server_key_version: "encryptionkey@0"
I20260925 19:02:51.814342 14223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:51.829339 14223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:51.831656 14223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:51.831817 14223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:51.832031 14223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:51.832329 14223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:51.832360 14223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:51.832381 14223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:51.832423 14223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:51.837849 14223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35451
I20260925 19:02:51.837914 14351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35451 every 8 connection(s)
I20260925 19:02:51.838228 14223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:02:51.843640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 14223
I20260925 19:02:51.843792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:02:51.843987 392 external_mini_cluster.cc:1428] Setting key c1a41fb6c4d965ef22f4c9bf9a2c2be2
Sep 25 19:02:51 dist-test-slave-49bz krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:51.847635 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:51.838985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52923 (local address 127.0.98.62:36379)
0925 19:02:51.839034 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:51.839036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:51.839208 (+ 172us) server_negotiation.cc:407] Connection header received
0925 19:02:51.839726 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:51.839727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:51.839739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:51.839804 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:51.840727 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:51.841221 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:51.842134 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:51.842412 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:51.842466 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:51.844900 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:51.844901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:51.844902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:51.844909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:51.846202 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:51.846650 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:51.846652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:51.846652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:51.846718 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:51.846975 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:51.846976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:51.846976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:51.847129 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:51.847186 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:51.847504 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 19:02:51.847573 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:02:51.847915 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:51.838937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:52923)
0925 19:02:51.839084 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:51.839124 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:51.839594 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:51.839858 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:51.839861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:51.840438 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:51.840618 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:51.840623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:51.841303 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:51.841304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:51.842057 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:51.842059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:51.842482 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:51.842482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:51.842581 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:51.843081 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:51.843089 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:51.844804 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:51.846333 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:51.846340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:51.846343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:51.846579 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:51.846777 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:51.846778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:51.846779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:51.846903 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:51.847197 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:51.847209 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:51.847371 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 19:02:51.847482 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 19:02:51.847651 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 19:02:51.848443 14352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260925 19:02:51.848542 14352 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:51.848709 14352 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260925 19:02:51.849227 14161 ts_manager.cc:194] Registered new tserver with Master: bb812bd1639640c0b625ba6b716c537a (127.0.98.1:35451)
I20260925 19:02:51.849848 14161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52923
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:02:51.868268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:51.949010 14357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:51.949249 14357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:51.949290 14357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:51.950417 14357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:51.950475 14357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:51.950515 14357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:02:51.951887 14357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:51.952142 14357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:51.952369 14357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:51.957072 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:51.957115 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:51.957217 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:51.957353 14357 server_base.cc:1034] running on GCE node
I20260925 19:02:51.957569 14357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:51.957824 14357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:51.958979 14357 hybrid_clock.cc:648] HybridClock initialized: now 1790362971958970 us; error 40 us; skew 500 ppm
Sep 25 19:02:51 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:51.961474 14357 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:02:51.961956 14357 webserver.cc:460] Webserver started at http://127.0.98.2:38739/ using document root <none> and password file <none>
I20260925 19:02:51.962294 14357 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:51.962373 14357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:51.962510 14357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:02:51.963557 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "0c92fabf617e429bb8979d6c1d9f951d"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "3d469879d57f66062196654fe44fa63e"
server_key_iv: "0f22f16c3189c60606189e5378818a10"
server_key_version: "encryptionkey@0"
I20260925 19:02:51.963912 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0c92fabf617e429bb8979d6c1d9f951d"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "3d469879d57f66062196654fe44fa63e"
server_key_iv: "0f22f16c3189c60606189e5378818a10"
server_key_version: "encryptionkey@0"
I20260925 19:02:51.965211 14357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:02:51.965917 14372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:51.966080 14357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:02:51.966174 14357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "0c92fabf617e429bb8979d6c1d9f951d"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "3d469879d57f66062196654fe44fa63e"
server_key_iv: "0f22f16c3189c60606189e5378818a10"
server_key_version: "encryptionkey@0"
I20260925 19:02:51.966306 14357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:51.981273 14357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:51.983669 14357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:51.983837 14357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:51.984061 14357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:51.984375 14357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:51.984426 14357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:51.984462 14357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:51.984490 14357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:51.990173 14357 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45319
I20260925 19:02:51.990232 14485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45319 every 8 connection(s)
I20260925 19:02:51.990587 14357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:02:51.992631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 14357
I20260925 19:02:51.992749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:02:51.992936 392 external_mini_cluster.cc:1428] Setting key 176cb253ff554c2c0bbc4f65ce658c14
Sep 25 19:02:51 dist-test-slave-49bz krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:52.001003 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:51.991294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50879 (local address 127.0.98.62:36379)
0925 19:02:51.991350 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:51.991352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:51.991557 (+ 205us) server_negotiation.cc:407] Connection header received
0925 19:02:51.992082 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:51.992083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:51.992096 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:51.992147 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:51.993390 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:51.993945 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:51.995395 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:51.995589 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:51.995674 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:51.998056 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:51.998058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:51.998059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:51.998067 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:51.999678 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.000038 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.000039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.000040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.000103 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.000341 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.000342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.000343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.000477 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:52.000551 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.000803 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.000895 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 19:02:52.001039 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:51.991265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:50879)
0925 19:02:51.991442 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:51.991475 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:51.991973 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:51.992214 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:51.992218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:51.993018 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:51.993283 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:51.993289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:51.994031 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:51.994032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:51.995309 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:51.995312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:51.995683 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:51.995684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:51.995819 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:51.996364 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:51.996373 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:51.997958 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:51.999754 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:51.999760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:51.999761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:51.999964 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:52.000170 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:52.000171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:52.000171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:52.000277 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:52.000545 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:52.000549 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:52.000663 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.000740 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.000878 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 19:02:52.001610 14486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260925 19:02:52.001708 14486 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:52.001912 14486 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260925 19:02:52.002353 14161 ts_manager.cc:194] Registered new tserver with Master: 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319)
I20260925 19:02:52.002677 14161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:02:52.019256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:52.096978 14491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:52.097178 14491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:52.097208 14491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:52.098275 14491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:52.098307 14491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:52.098331 14491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:02:52.099637 14491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:52.099817 14491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:52.099970 14491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:52.104679 14497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:52.104691 14499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:52.104679 14496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:52.105085 14491 server_base.cc:1034] running on GCE node
I20260925 19:02:52.105262 14491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:52.105445 14491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:52.106580 14491 hybrid_clock.cc:648] HybridClock initialized: now 1790362972106564 us; error 28 us; skew 500 ppm
Sep 25 19:02:52 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:52.108909 14491 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:02:52.109268 14491 webserver.cc:460] Webserver started at http://127.0.98.3:41305/ using document root <none> and password file <none>
I20260925 19:02:52.109440 14491 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:52.109479 14491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:52.109572 14491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:02:52.110553 14491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "7618c9e6485b4809be7593519e31ca2b"
format_stamp: "Formatted at 2026-09-25 19:02:52 on dist-test-slave-49bz"
server_key: "4933826bf2cd07adb06fd875c783c29d"
server_key_iv: "ce87c6625049b9b842d10c6c4822ceaf"
server_key_version: "encryptionkey@0"
I20260925 19:02:52.110846 14491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7618c9e6485b4809be7593519e31ca2b"
format_stamp: "Formatted at 2026-09-25 19:02:52 on dist-test-slave-49bz"
server_key: "4933826bf2cd07adb06fd875c783c29d"
server_key_iv: "ce87c6625049b9b842d10c6c4822ceaf"
server_key_version: "encryptionkey@0"
I20260925 19:02:52.112006 14491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:52.112685 14506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.112845 14491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:02:52.112943 14491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "7618c9e6485b4809be7593519e31ca2b"
format_stamp: "Formatted at 2026-09-25 19:02:52 on dist-test-slave-49bz"
server_key: "4933826bf2cd07adb06fd875c783c29d"
server_key_iv: "ce87c6625049b9b842d10c6c4822ceaf"
server_key_version: "encryptionkey@0"
I20260925 19:02:52.113018 14491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:52.122144 14491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:52.124048 14491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:52.124197 14491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:52.124406 14491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:52.124696 14491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:52.124774 14491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.124814 14491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:52.124843 14491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.130822 14491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44667
I20260925 19:02:52.130854 14619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44667 every 8 connection(s)
I20260925 19:02:52.131181 14491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:02:52.133618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 14491
I20260925 19:02:52.133754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:02:52.133936 392 external_mini_cluster.cc:1428] Setting key 6319a841d8e72d879a45f25feda9e8b7
Sep 25 19:02:52 dist-test-slave-49bz krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:52.140038 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.131861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52999 (local address 127.0.98.62:36379)
0925 19:02:52.131912 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.131914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.132128 (+ 214us) server_negotiation.cc:407] Connection header received
0925 19:02:52.132593 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.132594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.132604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.132649 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:52.133549 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.134244 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.135303 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.135514 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.135597 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.137694 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:52.137695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:52.137696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:52.137702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:52.138997 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.139339 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.139340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.139340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.139386 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.139581 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.139581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.139582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.139669 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:52.139712 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.139924 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.139981 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:02:52.140224 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.131839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:52999)
0925 19:02:52.131984 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.132017 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.132511 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.132706 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.132709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.133245 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:52.133434 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.133439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.134362 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.134363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.135191 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.135193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.135606 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.135608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.135730 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.136289 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:52.136297 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:52.137616 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:52.139073 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:52.139078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:52.139079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:52.139279 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:52.139446 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:52.139447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:52.139447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:52.139533 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:52.139715 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:52.139718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:52.139826 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.139899 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.140004 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260925 19:02:52.140631 14620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260925 19:02:52.140698 14620 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:52.140861 14620 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260925 19:02:52.141222 14161 ts_manager.cc:194] Registered new tserver with Master: 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667)
I20260925 19:02:52.141607 14161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52999
I20260925 19:02:52.145427 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:02:52.152424 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.146795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.0.98.62:36379)
0925 19:02:52.146858 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.146860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.146954 (+ 94us) server_negotiation.cc:407] Connection header received
0925 19:02:52.147026 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.147027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.147036 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.147089 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:52.147655 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.148131 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.148855 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.149054 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.149123 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.150197 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:52.150199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:52.150199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:52.150206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:52.151312 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.151648 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.151649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.151649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.151706 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.151909 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.151910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.151910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.152046 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:52.152144 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.152250 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.152331 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:02:52.153709 14161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:02:52.155228 14161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:02:52.161321 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.157492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44667 (local address 127.0.0.1:55908)
0925 19:02:52.157630 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.157639 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.157680 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.158073 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.158074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.158076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:52.158109 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.158110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.158988 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.158988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.160138 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.160141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.160969 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.160970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.161089 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.161108 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.161182 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.161236 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":56,"threads_started":1}
I20260925 19:02:52.161335 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.157650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.0.98.3:44667)
0925 19:02:52.157873 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.157878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.157886 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:02:52.157919 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.157921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.157934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.158000 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:52.158180 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.158892 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.160233 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.160882 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.160969 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.161062 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.161209 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.161248 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260925 19:02:52.161522 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.157285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.0.1:33138)
0925 19:02:52.157453 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.157493 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.157554 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.157921 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.157924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.157926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:52.157990 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.157992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.159456 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.159457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.160356 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.160358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.161299 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.161300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.161357 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.161361 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.161389 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.161427 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20260925 19:02:52.161700 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.157410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.0.98.1:35451)
0925 19:02:52.157583 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.157588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.157717 (+ 129us) server_negotiation.cc:407] Connection header received
0925 19:02:52.157794 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.157796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.157824 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.157902 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:52.158141 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.159302 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.160613 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.161249 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.161327 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.161381 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.161561 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.161604 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260925 19:02:52.161892 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.157389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.0.98.2:45319)
0925 19:02:52.157688 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.157691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.157700 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:02:52.157736 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.157738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.157784 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.158236 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:52.158324 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.159324 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.160466 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.161189 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.161687 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.161749 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.161788 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.161826 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":73,"threads_started":1}
I20260925 19:02:52.161865 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.157295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35451 (local address 127.0.0.1:41836)
0925 19:02:52.157590 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.157600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.157653 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.157982 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.157983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.157986 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:52.158047 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.158049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.159432 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.159433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.160536 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.160538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.161408 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.161409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.161454 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.161456 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.161497 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.161524 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":40,"thread_start_us":77,"threads_started":1}
I20260925 19:02:52.162163 14554 tablet_service.cc:1467] Processing CreateTablet for tablet 805b1bd0db494c8dbc72c43d75dc1f75 (DEFAULT_TABLE table=test-table [id=783ebb13ecb04f64ae26c23674278444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:52.162571 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805b1bd0db494c8dbc72c43d75dc1f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:52.162556 14420 tablet_service.cc:1467] Processing CreateTablet for tablet 805b1bd0db494c8dbc72c43d75dc1f75 (DEFAULT_TABLE table=test-table [id=783ebb13ecb04f64ae26c23674278444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:52.162647 14286 tablet_service.cc:1467] Processing CreateTablet for tablet 805b1bd0db494c8dbc72c43d75dc1f75 (DEFAULT_TABLE table=test-table [id=783ebb13ecb04f64ae26c23674278444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:52.162858 14420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805b1bd0db494c8dbc72c43d75dc1f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:52.162937 14286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805b1bd0db494c8dbc72c43d75dc1f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:52.165189 14634 tablet_bootstrap.cc:492] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: Bootstrap starting.
I20260925 19:02:52.165855 14634 tablet_bootstrap.cc:654] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:52.165827 14635 tablet_bootstrap.cc:492] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: Bootstrap starting.
I20260925 19:02:52.166412 14636 tablet_bootstrap.cc:492] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Bootstrap starting.
I20260925 19:02:52.166530 14635 tablet_bootstrap.cc:654] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:52.166595 14634 log.cc:824] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:52.167024 14635 log.cc:824] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:52.167055 14636 tablet_bootstrap.cc:654] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:52.167379 14634 tablet_bootstrap.cc:492] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: No bootstrap required, opened a new log
I20260925 19:02:52.167457 14634 ts_tablet_manager.cc:1397] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:02:52.167565 14636 log.cc:824] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:52.167793 14635 tablet_bootstrap.cc:492] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: No bootstrap required, opened a new log
I20260925 19:02:52.167855 14635 ts_tablet_manager.cc:1397] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:02:52.168296 14636 tablet_bootstrap.cc:492] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: No bootstrap required, opened a new log
I20260925 19:02:52.168363 14636 ts_tablet_manager.cc:1397] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:02:52.169332 14634 raft_consensus.cc:348] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.169538 14634 raft_consensus.cc:374] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:52.169571 14634 raft_consensus.cc:729] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c92fabf617e429bb8979d6c1d9f951d, State: Initialized, Role: FOLLOWER
I20260925 19:02:52.169737 14634 consensus_queue.cc:260] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.170003 14634 ts_tablet_manager.cc:1428] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:02:52.169862 14635 raft_consensus.cc:348] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.170017 14486 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
I20260925 19:02:52.170061 14635 raft_consensus.cc:374] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:52.169934 14636 raft_consensus.cc:348] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.170095 14635 raft_consensus.cc:729] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7618c9e6485b4809be7593519e31ca2b, State: Initialized, Role: FOLLOWER
I20260925 19:02:52.170099 14636 raft_consensus.cc:374] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:52.170131 14636 raft_consensus.cc:729] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb812bd1639640c0b625ba6b716c537a, State: Initialized, Role: FOLLOWER
I20260925 19:02:52.170212 14635 consensus_queue.cc:260] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.170274 14636 consensus_queue.cc:260] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.170513 14635 ts_tablet_manager.cc:1428] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:02:52.170533 14636 ts_tablet_manager.cc:1428] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:02:52.170650 14620 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
I20260925 19:02:52.170912 14352 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
I20260925 19:02:52.236722 14642 raft_consensus.cc:482] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:02:52.236838 14642 raft_consensus.cc:504] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.237250 14642 leader_election.cc:288] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319), 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667)
I20260925 19:02:52.239858 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.237452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.98.1:34911)
0925 19:02:52.237509 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.237521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.237587 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.237728 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.237729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.237731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:52.237805 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.237806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.238410 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.238411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.239070 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.239072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.239577 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.239578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.239674 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.239677 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.239744 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.239790 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:02:52.239857 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.237564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34911 (local address 127.0.98.2:45319)
0925 19:02:52.237612 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.237614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.237626 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:02:52.237655 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.237656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.237667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.237724 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:52.237874 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.238329 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.239157 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.239495 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.239578 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.239587 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.239748 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.239796 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:02:52.240134 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.237642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44715 (local address 127.0.98.3:44667)
0925 19:02:52.237685 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.237687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.237760 (+ 73us) server_negotiation.cc:407] Connection header received
0925 19:02:52.237819 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.237820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.237832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.237878 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:52.238022 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.238516 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.239339 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.239705 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.239797 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.239807 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.240004 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.240063 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:02:52.240190 14440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805b1bd0db494c8dbc72c43d75dc1f75" candidate_uuid: "bb812bd1639640c0b625ba6b716c537a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c92fabf617e429bb8979d6c1d9f951d" is_pre_election: true
I20260925 19:02:52.240362 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.237567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44667 (local address 127.0.98.1:44715)
0925 19:02:52.237700 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.237709 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.237768 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.237879 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.237880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.237881 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:52.237949 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.237950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.238599 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.238600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.239257 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.239259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.239810 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.239812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.239895 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.239898 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.240241 (+ 343us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.240285 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260925 19:02:52.240382 14440 raft_consensus.cc:2454] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb812bd1639640c0b625ba6b716c537a in term 0.
I20260925 19:02:52.240620 14240 leader_election.cc:302] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c92fabf617e429bb8979d6c1d9f951d, bb812bd1639640c0b625ba6b716c537a; no voters:
I20260925 19:02:52.240659 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805b1bd0db494c8dbc72c43d75dc1f75" candidate_uuid: "bb812bd1639640c0b625ba6b716c537a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7618c9e6485b4809be7593519e31ca2b" is_pre_election: true
I20260925 19:02:52.240744 14642 raft_consensus.cc:2781] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:02:52.240793 14642 raft_consensus.cc:482] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:02:52.240823 14642 raft_consensus.cc:3037] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:52.240837 14574 raft_consensus.cc:2454] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb812bd1639640c0b625ba6b716c537a in term 0.
W20260925 19:02:52.241120 14487 tablet.cc:2369] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:02:52.241663 14642 raft_consensus.cc:504] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.241804 14642 leader_election.cc:288] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 1 election: Requested vote from peers 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319), 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667)
I20260925 19:02:52.241957 14440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805b1bd0db494c8dbc72c43d75dc1f75" candidate_uuid: "bb812bd1639640c0b625ba6b716c537a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c92fabf617e429bb8979d6c1d9f951d"
I20260925 19:02:52.241979 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805b1bd0db494c8dbc72c43d75dc1f75" candidate_uuid: "bb812bd1639640c0b625ba6b716c537a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7618c9e6485b4809be7593519e31ca2b"
I20260925 19:02:52.242039 14574 raft_consensus.cc:3037] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:52.242060 14440 raft_consensus.cc:3037] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:52.242795 14574 raft_consensus.cc:2454] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb812bd1639640c0b625ba6b716c537a in term 1.
I20260925 19:02:52.242870 14440 raft_consensus.cc:2454] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb812bd1639640c0b625ba6b716c537a in term 1.
I20260925 19:02:52.242949 14240 leader_election.cc:302] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7618c9e6485b4809be7593519e31ca2b, bb812bd1639640c0b625ba6b716c537a; no voters:
I20260925 19:02:52.243041 14642 raft_consensus.cc:2781] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:02:52.243176 14642 raft_consensus.cc:686] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 1 LEADER]: Becoming Leader. State: Replica: bb812bd1639640c0b625ba6b716c537a, State: Running, Role: LEADER
I20260925 19:02:52.243283 14642 consensus_queue.cc:237] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.244057 14161 catalog_manager.cc:5495] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb812bd1639640c0b625ba6b716c537a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bb812bd1639640c0b625ba6b716c537a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } health_report { overall_health: HEALTHY } } }
I20260925 19:02:52.259974 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.253622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.0.98.1:35451)
0925 19:02:52.253686 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.253694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.253729 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:02:52.253955 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.253955 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.254002 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.254626 (+ 624us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:02:52.254977 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.256460 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.259250 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.259423 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.259485 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.259634 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:02:52.259747 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:02:52.259808 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.259848 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.259865 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:02:52.261855 14574 raft_consensus.cc:1261] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b [term 1 FOLLOWER]: Refusing update from remote peer bb812bd1639640c0b625ba6b716c537a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:02:52.261889 14440 raft_consensus.cc:1261] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d [term 1 FOLLOWER]: Refusing update from remote peer bb812bd1639640c0b625ba6b716c537a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:02:52.262259 14644 consensus_queue.cc:1035] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:02:52.262431 14642 consensus_queue.cc:1035] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:02:52.264907 14649 mvcc.cc:205] Tried to move back new op lower bound from 7333326734381400064 to 7333326734308909056. Current Snapshot: MvccSnapshot[applied={T|T < 7333326734381400064}]
I20260925 19:02:52.267037 14161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:02:52.267125 14161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:02:52.267881 14161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Sending DeleteTablet for 3 replicas of tablet 805b1bd0db494c8dbc72c43d75dc1f75
I20260925 19:02:52.268234 14554 tablet_service.cc:1514] Processing DeleteTablet for tablet 805b1bd0db494c8dbc72c43d75dc1f75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55908
I20260925 19:02:52.268245 14420 tablet_service.cc:1514] Processing DeleteTablet for tablet 805b1bd0db494c8dbc72c43d75dc1f75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33138
I20260925 19:02:52.268260 14286 tablet_service.cc:1514] Processing DeleteTablet for tablet 805b1bd0db494c8dbc72c43d75dc1f75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41836
I20260925 19:02:52.268456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 14223
I20260925 19:02:52.269769 14658 tablet_replica.cc:330] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: stopping tablet replica
I20260925 19:02:52.269876 14658 raft_consensus.cc:2229] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:02:52.269989 14658 raft_consensus.cc:2258] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:02:52.270134 14657 tablet_replica.cc:330] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: stopping tablet replica
I20260925 19:02:52.270232 14657 raft_consensus.cc:2229] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:02:52.270349 14657 raft_consensus.cc:2258] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:02:52.270816 14657 ts_tablet_manager.cc:1905] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:02:52.270854 14658 ts_tablet_manager.cc:1905] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:02:52.272287 14657 ts_tablet_manager.cc:1918] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:02:52.272287 14658 ts_tablet_manager.cc:1918] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:02:52.272364 14657 log.cc:1196] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/805b1bd0db494c8dbc72c43d75dc1f75
I20260925 19:02:52.272364 14658 log.cc:1196] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/805b1bd0db494c8dbc72c43d75dc1f75
I20260925 19:02:52.272660 14657 ts_tablet_manager.cc:1939] T 805b1bd0db494c8dbc72c43d75dc1f75 P 7618c9e6485b4809be7593519e31ca2b: Deleting consensus metadata
I20260925 19:02:52.272660 14658 ts_tablet_manager.cc:1939] T 805b1bd0db494c8dbc72c43d75dc1f75 P 0c92fabf617e429bb8979d6c1d9f951d: Deleting consensus metadata
I20260925 19:02:52.273036 14146 catalog_manager.cc:4841] TS 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667): tablet 805b1bd0db494c8dbc72c43d75dc1f75 (table test-table [id=783ebb13ecb04f64ae26c23674278444]) successfully deleted
I20260925 19:02:52.273157 14146 catalog_manager.cc:4841] TS 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319): tablet 805b1bd0db494c8dbc72c43d75dc1f75 (table test-table [id=783ebb13ecb04f64ae26c23674278444]) successfully deleted
W20260925 19:02:52.275578 14145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35451 (error 108)
I20260925 19:02:52.275718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 14357
I20260925 19:02:52.276697 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.276558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35451 (local address 127.0.0.1:41862)
0925 19:02:52.276601 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.276659 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 19:02:52.276839 14145 catalog_manager.cc:4584] TS bb812bd1639640c0b625ba6b716c537a (127.0.98.1:35451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 805b1bd0db494c8dbc72c43d75dc1f75: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35451: connect: Connection refused (error 111)
I20260925 19:02:52.280776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 14491
I20260925 19:02:52.285517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 14129
I20260925 19:02:52.290880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36379
--webserver_interface=127.0.98.62
--webserver_port=43935
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:52.369249 14661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:52.369442 14661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:52.369463 14661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:52.370608 14661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:02:52.370649 14661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:52.370663 14661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:52.370676 14661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:02:52.370688 14661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:02:52.372074 14661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:52.372303 14661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:52.372545 14661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:52.380951 14670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:52.380951 14668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:52.380965 14667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:52.381141 14661 server_base.cc:1034] running on GCE node
I20260925 19:02:52.381456 14661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:52.381781 14661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:52.382933 14661 hybrid_clock.cc:648] HybridClock initialized: now 1790362972382929 us; error 35 us; skew 500 ppm
Sep 25 19:02:52 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:52.392036 14661 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:02:52.392552 14661 webserver.cc:460] Webserver started at http://127.0.98.62:43935/ using document root <none> and password file <none>
I20260925 19:02:52.392797 14661 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:52.392877 14661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:52.394371 14661 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:02:52.395119 14677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.395306 14661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:52.395408 14661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "34031c0838174e07ac6f48a64d042ce2"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "75bc3acce9d83fdab21bfbf4322accea"
server_key_iv: "ee71fc90c9679513ebbac245a479906f"
server_key_version: "encryptionkey@0"
I20260925 19:02:52.395825 14661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:52.416625 14661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:52.419132 14661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:52.419314 14661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:52.423296 14661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36379
I20260925 19:02:52.423334 14729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36379 every 8 connection(s)
I20260925 19:02:52.423722 14661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:02:52.425319 14730 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:02:52.425438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 14661
I20260925 19:02:52.425724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35451
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36379
--webserver_port=45147
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:02:52.426354 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Bootstrap starting.
I20260925 19:02:52.434748 14730 log.cc:824] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:52.436612 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:02:52.436902 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Bootstrap complete.
I20260925 19:02:52.439301 14730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:52.439523 14730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34031c0838174e07ac6f48a64d042ce2, State: Initialized, Role: FOLLOWER
I20260925 19:02:52.439697 14730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:52.439785 14730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:02:52.439842 14730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:02:52.439913 14730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:02:52.440937 14730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:52.441056 14730 leader_election.cc:302] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34031c0838174e07ac6f48a64d042ce2; no voters:
I20260925 19:02:52.441249 14730 leader_election.cc:288] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:02:52.441287 14734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:02:52.441398 14734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 2 LEADER]: Becoming Leader. State: Replica: 34031c0838174e07ac6f48a64d042ce2, State: Running, Role: LEADER
I20260925 19:02:52.441483 14734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:52.441522 14730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:02:52.441731 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34031c0838174e07ac6f48a64d042ce2. Latest consensus state: current_term: 2 leader_uuid: "34031c0838174e07ac6f48a64d042ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } }
I20260925 19:02:52.441811 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:52.441735 14735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "34031c0838174e07ac6f48a64d042ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } }
I20260925 19:02:52.442008 14735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:52.442020 14738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:02:52.442914 14738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:02:52.443017 14738 catalog_manager.cc:1259] Loaded cluster ID: 2274be48efeb41f8822a473c0dc044fd
I20260925 19:02:52.443053 14738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:02:52.443277 14738 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 19:02:52.448096 14738 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 19:02:52.450518 14738 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:02:52.450742 14738 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Loaded TSK: 0
I20260925 19:02:52.450980 14738 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:02:52.507324 14732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:52.507540 14732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:52.507578 14732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:52.508657 14732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:52.508719 14732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:52.508775 14732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:02:52.510206 14732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45147
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:52.510448 14732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:52.510660 14732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:52.516424 14757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:52.516422 14760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:52.516458 14758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:52.516712 14732 server_base.cc:1034] running on GCE node
I20260925 19:02:52.516907 14732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:52.517174 14732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:52.518342 14732 hybrid_clock.cc:648] HybridClock initialized: now 1790362972518319 us; error 32 us; skew 500 ppm
Sep 25 19:02:52 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:52.520707 14732 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:02:52.521150 14732 webserver.cc:460] Webserver started at http://127.0.98.1:45147/ using document root <none> and password file <none>
I20260925 19:02:52.521384 14732 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:52.521435 14732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:52.522799 14732 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:52.523501 14767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.523691 14732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:52.523778 14732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "bb812bd1639640c0b625ba6b716c537a"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "eb8e359ceef34fc508dee395b00601c8"
server_key_iv: "b39a31b7b4184253fa0643c42cf721c3"
server_key_version: "encryptionkey@0"
I20260925 19:02:52.524129 14732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:52.538834 14732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:52.541005 14732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:52.541154 14732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:52.541368 14732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:52.541895 14774 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:02:52.543812 14732 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:02:52.543880 14732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:02:52.543928 14732 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:02:52.544584 14732 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:02:52.544641 14732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:02:52.544708 14774 tablet_bootstrap.cc:492] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Bootstrap starting.
I20260925 19:02:52.552083 14732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35451
I20260925 19:02:52.552122 14881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35451 every 8 connection(s)
I20260925 19:02:52.552487 14732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:02:52.555137 14774 log.cc:824] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:52.557730 14774 tablet_bootstrap.cc:492] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:02:52.558061 14774 tablet_bootstrap.cc:492] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Bootstrap complete.
I20260925 19:02:52.559249 14774 ts_tablet_manager.cc:1397] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260925 19:02:52.560482 14774 raft_consensus.cc:348] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.560801 14774 raft_consensus.cc:729] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb812bd1639640c0b625ba6b716c537a, State: Initialized, Role: FOLLOWER
I20260925 19:02:52.560897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 14732
I20260925 19:02:52.560945 14774 consensus_queue.cc:260] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.561167 14774 ts_tablet_manager.cc:1428] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 19:02:52.561177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45319
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36379
--webserver_port=38739
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:02:52 dist-test-slave-49bz krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:52.569768 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.553147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:39467)
0925 19:02:52.555408 (+ 2261us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.555451 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.558418 (+ 2967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.558855 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.558859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.559112 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:52.559324 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.559331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.560471 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.560473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.561753 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.561756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.562712 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.562713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.562850 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.563485 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:52.563495 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:52.565370 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:52.567853 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:52.567860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:52.567862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:52.568180 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:52.568469 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:52.568469 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:52.568470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:52.568618 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:52.569072 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:52.569078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:52.569243 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.569360 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.569533 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2223,"thread_start_us":123,"threads_started":1}
I20260925 19:02:52.570087 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.553275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39467 (local address 127.0.98.62:36379)
0925 19:02:52.554588 (+ 1313us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.554595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.555588 (+ 993us) server_negotiation.cc:407] Connection header received
0925 19:02:52.558636 (+ 3048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.558642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.558686 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.558817 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:52.559486 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.560357 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.561859 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.562490 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.562594 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.565502 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:52.565507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:52.565507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:52.565519 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:52.567754 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.568275 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.568278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.568279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.568392 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.568688 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.568688 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.568689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.568991 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:52.569073 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.569447 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.569606 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":68,"threads_started":1}
I20260925 19:02:52.570684 14882 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260925 19:02:52.570789 14882 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:52.570974 14882 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260925 19:02:52.571683 14694 ts_manager.cc:194] Registered new tserver with Master: bb812bd1639640c0b625ba6b716c537a (127.0.98.1:35451)
I20260925 19:02:52.572813 14694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39467
I20260925 19:02:52.574059 14882 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
I20260925 19:02:52.578215 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.573170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35451 (local address 127.0.0.1:41872)
0925 19:02:52.573340 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.573381 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.573443 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.574526 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.574528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.574532 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:52.574595 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.574597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.575874 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.575875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.576969 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.576972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.577913 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.577914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.577990 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.578002 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.578076 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.578135 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260925 19:02:52.578212 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.573380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.0.98.1:35451)
0925 19:02:52.574360 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.574364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.574374 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:02:52.574405 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.574407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.574422 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.574669 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:52.574707 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.575781 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.577061 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.577818 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.577893 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.577945 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.578113 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.578148 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":64,"threads_started":1}
I20260925 19:02:52.578792 14815 tablet_service.cc:1514] Processing DeleteTablet for tablet 805b1bd0db494c8dbc72c43d75dc1f75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41872
I20260925 19:02:52.579358 14891 tablet_replica.cc:330] stopping tablet replica
I20260925 19:02:52.579474 14891 raft_consensus.cc:2229] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:02:52.579540 14891 raft_consensus.cc:2258] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:02:52.579872 14891 ts_tablet_manager.cc:1905] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:02:52.581851 14891 ts_tablet_manager.cc:1918] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:02:52.581945 14891 log.cc:1196] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/805b1bd0db494c8dbc72c43d75dc1f75
I20260925 19:02:52.582168 14891 ts_tablet_manager.cc:1939] T 805b1bd0db494c8dbc72c43d75dc1f75 P bb812bd1639640c0b625ba6b716c537a: Deleting consensus metadata
I20260925 19:02:52.582746 14678 catalog_manager.cc:4841] TS bb812bd1639640c0b625ba6b716c537a (127.0.98.1:35451): tablet 805b1bd0db494c8dbc72c43d75dc1f75 (table test-table [id=783ebb13ecb04f64ae26c23674278444]) successfully deleted
W20260925 19:02:52.643572 14888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:52.643752 14888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:52.643774 14888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:52.644881 14888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:52.644919 14888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:52.644972 14888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:02:52.646603 14888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38739
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:52.646823 14888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:52.647033 14888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:52.651727 14899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:52.651788 14897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:52.651841 14896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:52.651968 14888 server_base.cc:1034] running on GCE node
I20260925 19:02:52.652184 14888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:52.652539 14888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:52.653792 14888 hybrid_clock.cc:648] HybridClock initialized: now 1790362972653778 us; error 24 us; skew 500 ppm
Sep 25 19:02:52 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:52.656339 14888 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:02:52.656764 14888 webserver.cc:460] Webserver started at http://127.0.98.2:38739/ using document root <none> and password file <none>
I20260925 19:02:52.656941 14888 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:52.656991 14888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:52.658195 14888 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:52.658779 14906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.658955 14888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.659046 14888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "0c92fabf617e429bb8979d6c1d9f951d"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "3d469879d57f66062196654fe44fa63e"
server_key_iv: "0f22f16c3189c60606189e5378818a10"
server_key_version: "encryptionkey@0"
I20260925 19:02:52.659353 14888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:52.676424 14888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:52.678081 14888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:52.678182 14888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:52.678433 14888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:52.678678 14888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:52.678706 14888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.678726 14888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:52.678738 14888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.684604 14888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45319
I20260925 19:02:52.684655 15019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45319 every 8 connection(s)
I20260925 19:02:52.684931 14888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:02:52.686232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 14888
I20260925 19:02:52.686573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44667
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36379
--webserver_port=41305
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:02:52 dist-test-slave-49bz krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:52.694657 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.685555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44295 (local address 127.0.98.62:36379)
0925 19:02:52.685615 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.685617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.685856 (+ 239us) server_negotiation.cc:407] Connection header received
0925 19:02:52.686595 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.686597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.686609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.686681 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:52.688112 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.688654 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.689598 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.689816 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.689896 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.691938 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:52.691940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:52.691940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:52.691947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:52.693295 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.693683 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.693684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.693684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.693750 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.693993 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.693994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.693995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.694113 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:52.694188 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.694483 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.694569 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:02:52.694840 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.685545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:44295)
0925 19:02:52.685708 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.685765 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.686473 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.686852 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.686857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.687670 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:52.687952 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.687960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.688741 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.688742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.689507 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.689508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.689909 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.689910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.690036 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.690588 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:52.690596 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:52.691830 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:52.693388 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:52.693395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:52.693397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:52.693609 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:52.693815 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:52.693816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:52.693817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:52.693929 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:52.694179 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:52.694185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:52.694355 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.694462 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.694606 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260925 19:02:52.695324 15020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260925 19:02:52.695438 15020 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:52.695616 15020 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260925 19:02:52.696012 14694 ts_manager.cc:194] Registered new tserver with Master: 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319)
I20260925 19:02:52.696597 14694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44295
W20260925 19:02:52.765789 15023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:52.765985 15023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:52.766008 15023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:52.767055 15023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:52.767088 15023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:52.767113 15023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:02:52.768419 15023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41305
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:52.768590 15023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:52.768750 15023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:52.773365 15028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:52.773433 15029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:52.773370 15031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:52.773569 15023 server_base.cc:1034] running on GCE node
I20260925 19:02:52.773757 15023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:52.773993 15023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:52.775177 15023 hybrid_clock.cc:648] HybridClock initialized: now 1790362972775164 us; error 30 us; skew 500 ppm
Sep 25 19:02:52 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:52.777635 15023 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:02:52.778092 15023 webserver.cc:460] Webserver started at http://127.0.98.3:41305/ using document root <none> and password file <none>
I20260925 19:02:52.778368 15023 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:52.778445 15023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:52.779726 15023 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:52.780301 15038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.780481 15023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:02:52.780577 15023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "7618c9e6485b4809be7593519e31ca2b"
format_stamp: "Formatted at 2026-09-25 19:02:52 on dist-test-slave-49bz"
server_key: "4933826bf2cd07adb06fd875c783c29d"
server_key_iv: "ce87c6625049b9b842d10c6c4822ceaf"
server_key_version: "encryptionkey@0"
I20260925 19:02:52.780920 15023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:52.793340 15023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:52.795559 15023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:52.795725 15023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:52.795938 15023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:52.796231 15023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:52.796283 15023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.796322 15023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:52.796350 15023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.802888 15023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44667
I20260925 19:02:52.802944 15151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44667 every 8 connection(s)
I20260925 19:02:52.803261 15023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:02:52 dist-test-slave-49bz krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:02:52.811570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 15023
I20260925 19:02:52.812052 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.803977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35245 (local address 127.0.98.62:36379)
0925 19:02:52.804059 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.804062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.804220 (+ 158us) server_negotiation.cc:407] Connection header received
0925 19:02:52.804736 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.804737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.804750 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.804806 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:52.805701 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.806217 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.807236 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.807428 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.807477 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.809343 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:52.809345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:52.809345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:52.809351 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:52.810757 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.811132 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.811132 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.811133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.811183 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.811394 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.811394 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.811395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.811491 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:52.811574 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.811890 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.811966 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:02:52.812263 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.803943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:35245)
0925 19:02:52.804096 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.804131 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.804625 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.804859 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.804863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.805408 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:52.805595 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.805599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.806378 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.806378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.807162 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.807164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.807494 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.807495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.807584 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.808006 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:52.808014 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:52.809256 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:52.810834 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:52.810839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:52.810840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:52.811066 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:52.811240 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:52.811241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:52.811241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:52.811339 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:52.811569 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:52.811577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:52.811751 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.811857 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.812007 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260925 19:02:52.812736 15152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260925 19:02:52.812849 15152 heartbeater.cc:464] Registering TS with master...
I20260925 19:02:52.813038 15152 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260925 19:02:52.813432 14694 ts_manager.cc:194] Registered new tserver with Master: 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667)
I20260925 19:02:52.813772 14694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35245
I20260925 19:02:52.818243 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.811956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.0.98.62:36379)
0925 19:02:52.812065 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.812067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.812216 (+ 149us) server_negotiation.cc:407] Connection header received
0925 19:02:52.812289 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.812290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.812301 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.812354 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:52.813183 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.813858 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.814845 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.815028 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.815082 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.816100 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:52.816101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:52.816101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:52.816107 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:52.817364 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.817655 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.817656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.817656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.817703 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:52.817855 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:52.817855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:52.817855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:52.817940 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:52.818053 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.818099 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.818167 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":42,"threads_started":1}
I20260925 19:02:52.818928 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:02:52.821165 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.819249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.0.98.62:36379)
0925 19:02:52.819294 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.819296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.819376 (+ 80us) server_negotiation.cc:407] Connection header received
0925 19:02:52.819419 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.819420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.819443 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.819501 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:02:52.819562 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.819970 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.820610 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.820782 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.820831 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.820946 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:02:52.821024 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:02:52.821078 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.821111 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.821126 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 19:02:52.821485 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:02:52.823146 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:02:52.825209 14815 tablet_service.cc:1467] Processing CreateTablet for tablet 661e96f0a71b42c8ab642c6fcde34bca (DEFAULT_TABLE table=test-table [id=d88d08fe3d27444480a761960b7b3d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:52.825419 14815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661e96f0a71b42c8ab642c6fcde34bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:52.827044 14774 tablet_bootstrap.cc:492] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Bootstrap starting.
I20260925 19:02:52.827288 14774 tablet_bootstrap.cc:654] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:52.827852 14774 tablet_bootstrap.cc:492] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: No bootstrap required, opened a new log
I20260925 19:02:52.827919 14774 ts_tablet_manager.cc:1397] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 19:02:52.828029 14774 raft_consensus.cc:348] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.828125 14774 raft_consensus.cc:374] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:52.828171 14774 raft_consensus.cc:729] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb812bd1639640c0b625ba6b716c537a, State: Initialized, Role: FOLLOWER
I20260925 19:02:52.828212 14774 consensus_queue.cc:260] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.828290 14774 ts_tablet_manager.cc:1428] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:52.829079 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.825046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44667 (local address 127.0.0.1:55914)
0925 19:02:52.825189 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.825198 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.825245 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.825612 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.825613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.825615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:52.825656 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.825658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.826761 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.826762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.827913 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.827915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.828840 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.828841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.828913 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.828916 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.828961 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.829007 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260925 19:02:52.829113 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.825278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.0.98.2:45319)
0925 19:02:52.825463 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.825468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.825479 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:02:52.825518 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.825520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.825536 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.825752 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:52.825828 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.826896 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.828099 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.828800 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.828891 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.828966 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.829028 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.829056 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260925 19:02:52.829138 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.824939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.0.1:33148)
0925 19:02:52.825002 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:52.825014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:52.825059 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:52.825642 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:52.825643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:52.825645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:52.825683 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.825684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.826999 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.827000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.828012 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:52.828014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.828898 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.828898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:52.828966 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.828968 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:02:52.829018 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:02:52.829049 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 19:02:52.829399 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:52.825291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.0.98.3:44667)
0925 19:02:52.825453 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:52.825457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:52.825467 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:02:52.825506 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:52.825507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:52.825519 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:52.825615 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:52.825713 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.826667 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.828006 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:52.828746 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:52.829190 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:52.829259 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:52.829304 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 19:02:52.829339 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260925 19:02:52.829784 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 661e96f0a71b42c8ab642c6fcde34bca (DEFAULT_TABLE table=test-table [id=d88d08fe3d27444480a761960b7b3d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:52.830005 15086 tablet_service.cc:1467] Processing CreateTablet for tablet 661e96f0a71b42c8ab642c6fcde34bca (DEFAULT_TABLE table=test-table [id=d88d08fe3d27444480a761960b7b3d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:02:52.830083 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661e96f0a71b42c8ab642c6fcde34bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:52.830325 15086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661e96f0a71b42c8ab642c6fcde34bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:02:52.832646 15162 tablet_bootstrap.cc:492] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Bootstrap starting.
I20260925 19:02:52.832952 15163 tablet_bootstrap.cc:492] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: Bootstrap starting.
I20260925 19:02:52.833222 15162 tablet_bootstrap.cc:654] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:52.833509 15163 tablet_bootstrap.cc:654] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: Neither blocks nor log segments found. Creating new log.
I20260925 19:02:52.833678 15162 log.cc:824] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:52.834079 15163 log.cc:824] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:52.834465 15162 tablet_bootstrap.cc:492] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: No bootstrap required, opened a new log
I20260925 19:02:52.834542 15162 ts_tablet_manager.cc:1397] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:02:52.834758 15163 tablet_bootstrap.cc:492] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: No bootstrap required, opened a new log
I20260925 19:02:52.834825 15163 ts_tablet_manager.cc:1397] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:02:52.836068 15162 raft_consensus.cc:348] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.836212 15162 raft_consensus.cc:374] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:52.836253 15162 raft_consensus.cc:729] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c92fabf617e429bb8979d6c1d9f951d, State: Initialized, Role: FOLLOWER
I20260925 19:02:52.836406 15162 consensus_queue.cc:260] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.836473 15163 raft_consensus.cc:348] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.836654 15163 raft_consensus.cc:374] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:02:52.836686 15162 ts_tablet_manager.cc:1428] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:02:52.836702 15163 raft_consensus.cc:729] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7618c9e6485b4809be7593519e31ca2b, State: Initialized, Role: FOLLOWER
I20260925 19:02:52.836695 15020 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
I20260925 19:02:52.836819 15163 consensus_queue.cc:260] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:52.837077 15163 ts_tablet_manager.cc:1428] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:02:52.837101 15152 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
W20260925 19:02:52.935520 15021 tablet.cc:2369] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:02:53.053882 15153 tablet.cc:2369] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:02:53.054865 14883 tablet.cc:2369] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:02:53.075402 15166 raft_consensus.cc:482] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:02:53.075517 15166 raft_consensus.cc:504] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:53.075825 15166 leader_election.cc:288] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667), bb812bd1639640c0b625ba6b716c537a (127.0.98.1:35451)
I20260925 19:02:53.078701 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.076170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43119 (local address 127.0.98.3:44667)
0925 19:02:53.076209 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:53.076211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:53.076423 (+ 212us) server_negotiation.cc:407] Connection header received
0925 19:02:53.076452 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:53.076453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:53.076468 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:53.076521 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:53.076688 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:53.077152 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:53.078036 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:53.078386 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:53.078438 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:53.078445 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:53.078592 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 19:02:53.078637 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:02:53.078769 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.076159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58145 (local address 127.0.98.1:35451)
0925 19:02:53.076209 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:53.076211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:53.076282 (+ 71us) server_negotiation.cc:407] Connection header received
0925 19:02:53.076313 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:53.076314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:53.076328 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:53.076385 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:02:53.076567 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:53.077069 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:53.078006 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:53.078408 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:53.078485 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:53.078494 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:53.078661 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 19:02:53.078697 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:02:53.078797 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.076085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35451 (local address 127.0.98.2:58145)
0925 19:02:53.076150 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.076164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:53.076208 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:53.076411 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:53.076412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:53.076414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:53.076490 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:53.076492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:53.077148 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:53.077149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:53.077915 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:53.077916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:53.078510 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:53.078511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:53.078589 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:53.078592 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:02:53.078643 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:02:53.078678 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":30}
I20260925 19:02:53.078697 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.076085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44667 (local address 127.0.98.2:43119)
0925 19:02:53.076292 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.076303 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:53.076373 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:53.076553 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:53.076554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:53.076556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:02:53.076621 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:53.076623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:53.077229 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:53.077229 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:53.077932 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:53.077934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:53.078458 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:53.078459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:53.078523 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:53.078528 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:02:53.078570 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 19:02:53.078597 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":14,"thread_start_us":96,"threads_started":1}
I20260925 19:02:53.079150 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661e96f0a71b42c8ab642c6fcde34bca" candidate_uuid: "0c92fabf617e429bb8979d6c1d9f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb812bd1639640c0b625ba6b716c537a" is_pre_election: true
I20260925 19:02:53.079330 14836 raft_consensus.cc:2454] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c92fabf617e429bb8979d6c1d9f951d in term 0.
I20260925 19:02:53.079360 15100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661e96f0a71b42c8ab642c6fcde34bca" candidate_uuid: "0c92fabf617e429bb8979d6c1d9f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7618c9e6485b4809be7593519e31ca2b" is_pre_election: true
I20260925 19:02:53.079525 15100 raft_consensus.cc:2454] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c92fabf617e429bb8979d6c1d9f951d in term 0.
I20260925 19:02:53.079558 14907 leader_election.cc:302] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c92fabf617e429bb8979d6c1d9f951d, bb812bd1639640c0b625ba6b716c537a; no voters:
I20260925 19:02:53.079676 15166 raft_consensus.cc:2781] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:02:53.079734 15166 raft_consensus.cc:482] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:02:53.079763 15166 raft_consensus.cc:3037] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:53.080417 15166 raft_consensus.cc:504] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:53.080549 15166 leader_election.cc:288] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [CANDIDATE]: Term 1 election: Requested vote from peers 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667), bb812bd1639640c0b625ba6b716c537a (127.0.98.1:35451)
I20260925 19:02:53.080687 15100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661e96f0a71b42c8ab642c6fcde34bca" candidate_uuid: "0c92fabf617e429bb8979d6c1d9f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7618c9e6485b4809be7593519e31ca2b"
I20260925 19:02:53.080710 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661e96f0a71b42c8ab642c6fcde34bca" candidate_uuid: "0c92fabf617e429bb8979d6c1d9f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb812bd1639640c0b625ba6b716c537a"
I20260925 19:02:53.080765 15100 raft_consensus.cc:3037] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:53.080816 14836 raft_consensus.cc:3037] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:02:53.081431 15100 raft_consensus.cc:2454] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c92fabf617e429bb8979d6c1d9f951d in term 1.
I20260925 19:02:53.081501 14836 raft_consensus.cc:2454] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c92fabf617e429bb8979d6c1d9f951d in term 1.
I20260925 19:02:53.081624 14908 leader_election.cc:302] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c92fabf617e429bb8979d6c1d9f951d, 7618c9e6485b4809be7593519e31ca2b; no voters:
I20260925 19:02:53.081727 15166 raft_consensus.cc:2781] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:02:53.081844 15166 raft_consensus.cc:686] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 1 LEADER]: Becoming Leader. State: Replica: 0c92fabf617e429bb8979d6c1d9f951d, State: Running, Role: LEADER
I20260925 19:02:53.081931 15166 consensus_queue.cc:237] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:53.082798 14693 catalog_manager.cc:5495] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0c92fabf617e429bb8979d6c1d9f951d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } health_report { overall_health: UNKNOWN } } }
I20260925 19:02:53.112679 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.110487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.0.98.2:45319)
0925 19:02:53.110543 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:53.110545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:53.110553 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:02:53.110636 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:53.110636 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:53.110655 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:53.110723 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:02:53.110862 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:53.111292 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:53.111963 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:53.112122 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:53.112175 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:53.112329 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:02:53.112450 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:02:53.112509 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:53.112538 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 19:02:53.112554 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:02:53.114347 14836 raft_consensus.cc:1261] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Refusing update from remote peer 0c92fabf617e429bb8979d6c1d9f951d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:02:53.114377 15100 raft_consensus.cc:1261] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b [term 1 FOLLOWER]: Refusing update from remote peer 0c92fabf617e429bb8979d6c1d9f951d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:02:53.114672 15166 consensus_queue.cc:1035] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:02:53.114816 15169 consensus_queue.cc:1035] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:02:53.117048 15173 mvcc.cc:205] Tried to move back new op lower bound from 7333326737873690624 to 7333326737744093184. Current Snapshot: MvccSnapshot[applied={T|T < 7333326737873690624}]
I20260925 19:02:53.118480 15174 mvcc.cc:205] Tried to move back new op lower bound from 7333326737873690624 to 7333326737744093184. Current Snapshot: MvccSnapshot[applied={T|T < 7333326737873690624 or (T in {7333326737873690624})}]
I20260925 19:02:53.119148 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:02:53.119243 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:02:53.119959 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Sending DeleteTablet for 3 replicas of tablet 661e96f0a71b42c8ab642c6fcde34bca
I20260925 19:02:53.120241 14815 tablet_service.cc:1514] Processing DeleteTablet for tablet 661e96f0a71b42c8ab642c6fcde34bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41872
I20260925 19:02:53.120286 15086 tablet_service.cc:1514] Processing DeleteTablet for tablet 661e96f0a71b42c8ab642c6fcde34bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55914
I20260925 19:02:53.120432 15181 tablet_replica.cc:330] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: stopping tablet replica
I20260925 19:02:53.120488 15181 raft_consensus.cc:2229] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:02:53.120527 15181 raft_consensus.cc:2258] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:02:53.120537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 14732
I20260925 19:02:53.120522 14954 tablet_service.cc:1514] Processing DeleteTablet for tablet 661e96f0a71b42c8ab642c6fcde34bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:02:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33148
I20260925 19:02:53.126402 15183 tablet_replica.cc:330] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: stopping tablet replica
I20260925 19:02:53.126405 15182 tablet_replica.cc:330] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: stopping tablet replica
I20260925 19:02:53.126523 15183 raft_consensus.cc:2229] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:02:53.126523 15182 raft_consensus.cc:2229] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:02:53.126621 15183 raft_consensus.cc:2258] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:02:53.126632 15182 raft_consensus.cc:2258] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:02:53.127102 15182 ts_tablet_manager.cc:1905] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:02:53.127133 15183 ts_tablet_manager.cc:1905] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 19:02:53.127588 14678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35451 (error 108)
I20260925 19:02:53.128297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 14888
I20260925 19:02:53.130700 15182 ts_tablet_manager.cc:1918] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:02:53.130771 15182 log.cc:1196] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/661e96f0a71b42c8ab642c6fcde34bca
I20260925 19:02:53.131049 15182 ts_tablet_manager.cc:1939] T 661e96f0a71b42c8ab642c6fcde34bca P 7618c9e6485b4809be7593519e31ca2b: Deleting consensus metadata
I20260925 19:02:53.131544 14679 catalog_manager.cc:4841] TS 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667): tablet 661e96f0a71b42c8ab642c6fcde34bca (table test-table [id=d88d08fe3d27444480a761960b7b3d33]) successfully deleted
I20260925 19:02:53.131546 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.131395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35451 (local address 127.0.0.1:41874)
0925 19:02:53.131450 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.131511 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 19:02:53.131704 14678 catalog_manager.cc:4584] TS bb812bd1639640c0b625ba6b716c537a (127.0.98.1:35451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 661e96f0a71b42c8ab642c6fcde34bca: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35451: connect: Connection refused (error 111)
I20260925 19:02:53.133051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 15023
I20260925 19:02:53.133123 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.133023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.0.1:33180)
0925 19:02:53.133068 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.133091 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 19:02:53.137599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 14661
I20260925 19:02:53.142449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36379
--webserver_interface=127.0.98.62
--webserver_port=43935
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:02:53.218644 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:53.218837 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:53.218858 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:53.219835 15185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:02:53.219866 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:53.219879 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:53.219890 15185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:02:53.219900 15185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:02:53.221181 15185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:53.221366 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:53.221539 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:53.226148 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:53.226137 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:53.226387 15185 server_base.cc:1034] running on GCE node
W20260925 19:02:53.226132 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:53.226645 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:53.226936 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:53.228101 15185 hybrid_clock.cc:648] HybridClock initialized: now 1790362973228087 us; error 41 us; skew 500 ppm
Sep 25 19:02:53 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:53.230542 15185 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:02:53.230966 15185 webserver.cc:460] Webserver started at http://127.0.98.62:43935/ using document root <none> and password file <none>
I20260925 19:02:53.231137 15185 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:53.231174 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:53.232359 15185 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:53.232931 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:53.233100 15185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:02:53.233182 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "34031c0838174e07ac6f48a64d042ce2"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "75bc3acce9d83fdab21bfbf4322accea"
server_key_iv: "ee71fc90c9679513ebbac245a479906f"
server_key_version: "encryptionkey@0"
I20260925 19:02:53.233493 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:53.248036 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:53.249756 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:53.249858 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:53.253545 15185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36379
I20260925 19:02:53.253575 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36379 every 8 connection(s)
I20260925 19:02:53.253955 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:02:53.255519 15253 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:02:53.256182 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Bootstrap starting.
I20260925 19:02:53.256721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 15185
I20260925 19:02:53.257022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35451
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36379
--webserver_port=45147
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:02:53.264844 15253 log.cc:824] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:53.267143 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:02:53.267431 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2: Bootstrap complete.
I20260925 19:02:53.269519 15253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:53.269747 15253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34031c0838174e07ac6f48a64d042ce2, State: Initialized, Role: FOLLOWER
I20260925 19:02:53.269950 15253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:53.270042 15253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:02:53.270092 15253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:02:53.270140 15253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 2 FOLLOWER]: Advancing to term 3
I20260925 19:02:53.271157 15253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:53.271268 15253 leader_election.cc:302] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34031c0838174e07ac6f48a64d042ce2; no voters:
I20260925 19:02:53.271467 15253 leader_election.cc:288] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 19:02:53.271502 15257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 19:02:53.271611 15257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [term 3 LEADER]: Becoming Leader. State: Replica: 34031c0838174e07ac6f48a64d042ce2, State: Running, Role: LEADER
I20260925 19:02:53.271690 15257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260925 19:02:53.271716 15253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:02:53.271953 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "34031c0838174e07ac6f48a64d042ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } }
I20260925 19:02:53.272048 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:53.272228 15261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:02:53.273139 15261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:02:53.273185 15259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34031c0838174e07ac6f48a64d042ce2. Latest consensus state: current_term: 3 leader_uuid: "34031c0838174e07ac6f48a64d042ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34031c0838174e07ac6f48a64d042ce2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } }
I20260925 19:02:53.273253 15259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34031c0838174e07ac6f48a64d042ce2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:02:53.273406 15261 catalog_manager.cc:1259] Loaded cluster ID: 2274be48efeb41f8822a473c0dc044fd
I20260925 19:02:53.273716 15261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 19:02:53.274008 15261 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd2c72529ad google::LogMessage::Fail() at ??:0
@ 0x7fd2c725588a google::LogMessage::SendToLog() at ??:0
@ 0x7fd2c72524de google::LogMessage::Flush() at ??:0
@ 0x7fd2c7253a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd2cf1780f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd2cf1818b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd2c91200d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd2c9115954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd2c9f796db start_thread at ??:0
@ 0x7fd2c5d5061f clone at ??:0
W20260925 19:02:53.350050 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:53.350234 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:53.350310 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:53.351527 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:53.351565 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:53.351591 15255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:02:53.352991 15255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45147
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:53.353219 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:53.353451 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:53.358402 15290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:53.358506 15255 server_base.cc:1034] running on GCE node
W20260925 19:02:53.358378 15292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:53.358373 15289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:53.358729 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:53.358974 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:53.360155 15255 hybrid_clock.cc:648] HybridClock initialized: now 1790362973360140 us; error 30 us; skew 500 ppm
Sep 25 19:02:53 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:53.362700 15255 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:02:53.363155 15255 webserver.cc:460] Webserver started at http://127.0.98.1:45147/ using document root <none> and password file <none>
I20260925 19:02:53.363394 15255 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:53.363480 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:53.364869 15255 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:53.365551 15299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:53.365736 15255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:02:53.365852 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "bb812bd1639640c0b625ba6b716c537a"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "eb8e359ceef34fc508dee395b00601c8"
server_key_iv: "b39a31b7b4184253fa0643c42cf721c3"
server_key_version: "encryptionkey@0"
I20260925 19:02:53.366195 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:53.378718 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:53.380859 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:53.381007 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:53.381209 15255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:53.381625 15306 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:02:53.383577 15255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:02:53.383630 15255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:02:53.383682 15255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:02:53.384249 15255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:02:53.384298 15255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:02:53.384379 15306 tablet_bootstrap.cc:492] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Bootstrap starting.
I20260925 19:02:53.391345 15255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35451
I20260925 19:02:53.391384 15413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35451 every 8 connection(s)
I20260925 19:02:53.391736 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:02:53.392100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 15255
I20260925 19:02:53.392362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45319
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36379
--webserver_port=38739
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:02:53.395418 15306 log.cc:824] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:02:53.398152 15306 tablet_bootstrap.cc:492] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:02:53.398527 15306 tablet_bootstrap.cc:492] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Bootstrap complete.
I20260925 19:02:53.399710 15306 ts_tablet_manager.cc:1397] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.002s
I20260925 19:02:53.401086 15306 raft_consensus.cc:348] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:53.401424 15306 raft_consensus.cc:729] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb812bd1639640c0b625ba6b716c537a, State: Initialized, Role: FOLLOWER
I20260925 19:02:53.401597 15306 consensus_queue.cc:260] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:53.401855 15306 ts_tablet_manager.cc:1428] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260925 19:02:53.467235 15415 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:02:53.392445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:51211)
0925 19:02:53.394058 (+ 1613us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.394102 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:53.394755 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:53.467035 (+ 72280us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1568,"thread_start_us":90,"threads_started":1}
W20260925 19:02:53.467504 15414 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36379 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36379: Client connection negotiation failed: client connection to 127.0.98.62:36379: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:02:53.467893 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.467754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:50445)
0925 19:02:53.467820 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.467867 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 19:02:53.468241 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.468145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:39361)
0925 19:02:53.468193 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.468221 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 19:02:53.468384 15414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:02:53.473532 15416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:53.473737 15416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:53.473759 15416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:53.475013 15416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:53.475054 15416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:53.475080 15416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:02:53.476509 15416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38739
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:53.476744 15416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:53.476981 15416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:53.481628 15424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:53.481771 15425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:53.481629 15427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:53.481995 15416 server_base.cc:1034] running on GCE node
I20260925 19:02:53.482211 15416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:53.482558 15416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:53.483700 15416 hybrid_clock.cc:648] HybridClock initialized: now 1790362973483689 us; error 28 us; skew 500 ppm
Sep 25 19:02:53 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:53.486131 15416 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:02:53.486619 15416 webserver.cc:460] Webserver started at http://127.0.98.2:38739/ using document root <none> and password file <none>
I20260925 19:02:53.486850 15416 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:53.486920 15416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:53.488129 15416 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:02:53.488834 15434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:53.489001 15416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:02:53.489094 15416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "0c92fabf617e429bb8979d6c1d9f951d"
format_stamp: "Formatted at 2026-09-25 19:02:51 on dist-test-slave-49bz"
server_key: "3d469879d57f66062196654fe44fa63e"
server_key_iv: "0f22f16c3189c60606189e5378818a10"
server_key_version: "encryptionkey@0"
I20260925 19:02:53.489454 15416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:53.503665 15416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:53.505475 15416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:53.505626 15416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:53.505834 15416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:53.506315 15441 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260925 19:02:53.507642 15441 ts_tablet_manager.cc:1852] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 19:02:53.507894 15441 ts_tablet_manager.cc:1905] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:02:53.509963 15441 ts_tablet_manager.cc:1918] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:02:53.510063 15441 log.cc:1196] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/661e96f0a71b42c8ab642c6fcde34bca
I20260925 19:02:53.510401 15441 ts_tablet_manager.cc:1939] T 661e96f0a71b42c8ab642c6fcde34bca P 0c92fabf617e429bb8979d6c1d9f951d: Deleting consensus metadata
I20260925 19:02:53.510612 15416 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 19:02:53.510668 15416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 19:02:53.510711 15416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:53.510736 15416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:53.516619 15416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45319
I20260925 19:02:53.516688 15548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45319 every 8 connection(s)
I20260925 19:02:53.516968 15416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:02:53.517954 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.517614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:58783)
0925 19:02:53.517763 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.517830 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
W20260925 19:02:53.518291 15549 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36379 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36379: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260925 19:02:53.518455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 15416
I20260925 19:02:53.518589 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.518473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:37389)
0925 19:02:53.518539 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.518568 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 19:02:53.518745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44667
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36379
--webserver_port=41305
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:02:53.518905 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.518808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:39883)
0925 19:02:53.518848 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.518874 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 19:02:53.518982 15549 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:02:53.597370 15552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:02:53.597559 15552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:02:53.597581 15552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:02:53.598727 15552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:02:53.598764 15552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:02:53.598790 15552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:02:53.600212 15552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41305
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:02:53.600440 15552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:02:53.600658 15552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:02:53.605455 15560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:53.605474 15558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:02:53.605475 15557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:02:53.605800 15552 server_base.cc:1034] running on GCE node
I20260925 19:02:53.606022 15552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:02:53.606312 15552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:02:53.607468 15552 hybrid_clock.cc:648] HybridClock initialized: now 1790362973607453 us; error 31 us; skew 500 ppm
Sep 25 19:02:53 dist-test-slave-49bz krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:02:53.609929 15552 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:02:53.610446 15552 webserver.cc:460] Webserver started at http://127.0.98.3:41305/ using document root <none> and password file <none>
I20260925 19:02:53.610678 15552 fs_manager.cc:359] Metadata directory not provided
I20260925 19:02:53.610729 15552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:02:53.612048 15552 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 19:02:53.612622 15567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:53.612802 15552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:53.612908 15552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "7618c9e6485b4809be7593519e31ca2b"
format_stamp: "Formatted at 2026-09-25 19:02:52 on dist-test-slave-49bz"
server_key: "4933826bf2cd07adb06fd875c783c29d"
server_key_iv: "ce87c6625049b9b842d10c6c4822ceaf"
server_key_version: "encryptionkey@0"
I20260925 19:02:53.613255 15552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:02:53.642204 15552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:02:53.644140 15552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:02:53.644318 15552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:02:53.644553 15552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:02:53.644858 15552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:02:53.644939 15552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:53.644984 15552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:02:53.645015 15552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:02:53.650846 15552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44667
I20260925 19:02:53.650923 15680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44667 every 8 connection(s)
I20260925 19:02:53.651262 15552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:02:53.652288 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.651932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:41575)
0925 19:02:53.652073 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.652136 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
W20260925 19:02:53.652693 15681 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36379 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36379: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
I20260925 19:02:53.652940 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.652857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:52339)
0925 19:02:53.652898 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.652924 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 19:02:53.653199 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:53.653128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:37253)
0925 19:02:53.653166 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:53.653186 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 19:02:53.653309 15681 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:02:53.653421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 15552
I20260925 19:02:54.468978 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:54.468740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:39905)
0925 19:02:54.468879 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:54.468918 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260925 19:02:54.519548 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:54.519351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:48813)
0925 19:02:54.519472 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:54.519510 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260925 19:02:54.653906 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:54.653696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:59591)
0925 19:02:54.653825 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:54.653860 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260925 19:02:54.722965 15688 raft_consensus.cc:482] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:02:54.723078 15688 raft_consensus.cc:504] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:54.723356 15688 leader_election.cc:288] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319), 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667)
Sep 25 19:02:54 dist-test-slave-49bz krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 19:02:54 dist-test-slave-49bz krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 19:02:54.735726 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:54.723527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45319 (local address 127.0.98.1:37075)
0925 19:02:54.723589 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:54.723601 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:54.723665 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:54.724733 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:54.724736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:54.725285 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:54.725573 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:54.725581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:54.727317 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:54.727319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:54.728660 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:54.728663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:54.729510 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:54.729510 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:54.729640 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:54.730300 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:54.730304 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:54.731936 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:54.733954 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:54.733960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:54.733962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:54.734279 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:54.734558 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:54.734559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:54.734559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:54.734691 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:54.735149 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:54.735154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:54.735319 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 19:02:54.735435 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 19:02:54.735597 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":76}
I20260925 19:02:54.735754 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:54.723696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37075 (local address 127.0.98.2:45319)
0925 19:02:54.723827 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:54.723841 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:54.723869 (+ 28us) server_negotiation.cc:407] Connection header received
0925 19:02:54.724512 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:54.724515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:54.724558 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:54.724667 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:54.725708 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:54.727185 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:54.728763 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:54.729408 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:54.729523 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:54.732044 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:54.732048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:54.732049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:54.732061 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:54.733871 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:54.734404 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:54.734407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:54.734408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:54.734488 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:54.734756 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:54.734757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:54.734757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:54.735075 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:54.735158 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:54.735486 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 19:02:54.735630 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260925 19:02:54.735901 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:54.723653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44667 (local address 127.0.98.1:36933)
0925 19:02:54.723763 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:54.723773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:02:54.723819 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:02:54.724733 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:02:54.724736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:02:54.725288 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:02:54.725573 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:54.725581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:54.727328 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:54.727329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:54.728660 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:02:54.728663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:02:54.729508 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:02:54.729510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:02:54.729640 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:54.730246 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:02:54.730256 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:02:54.732428 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:02:54.734615 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:54.734616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:54.734616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:54.734834 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:54.735071 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:02:54.735072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:02:54.735072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:02:54.735141 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:02:54.735567 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:02:54.735568 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:02:54.735614 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 19:02:54.735691 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:02:54.735788 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":86,"thread_start_us":57,"threads_started":1}
I20260925 19:02:54.736038 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:54.723753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36933 (local address 127.0.98.3:44667)
0925 19:02:54.723900 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 19:02:54.723906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 19:02:54.723931 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:02:54.724434 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:02:54.724437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:02:54.724496 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:02:54.724655 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:02:54.725708 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:54.727174 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:54.728763 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:02:54.729407 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:02:54.729523 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:02:54.732529 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:02:54.732532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:02:54.732533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:02:54.732543 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:02:54.734539 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:54.734908 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:54.734911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:54.734911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:54.734984 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:02:54.735206 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:02:54.735206 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:02:54.735207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:02:54.735494 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:02:54.735571 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:02:54.735762 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 19:02:54.735904 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
W20260925 19:02:54.736826 15301 leader_election.cc:341] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319): Not found: Tablet not found: 661e96f0a71b42c8ab642c6fcde34bca
W20260925 19:02:54.736923 15301 leader_election.cc:341] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667): Not found: Tablet not found: 661e96f0a71b42c8ab642c6fcde34bca
I20260925 19:02:54.736965 15301 leader_election.cc:302] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb812bd1639640c0b625ba6b716c537a; no voters: 0c92fabf617e429bb8979d6c1d9f951d, 7618c9e6485b4809be7593519e31ca2b
I20260925 19:02:54.737099 15688 raft_consensus.cc:2726] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:02:55.469617 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:55.469410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:35025)
0925 19:02:55.469536 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:55.469571 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260925 19:02:55.520139 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:55.519965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:46579)
0925 19:02:55.520071 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:55.520103 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260925 19:02:55.654564 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:55.654385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:48273)
0925 19:02:55.654501 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:55.654530 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":53,"threads_started":1}
I20260925 19:02:56.445320 15695 raft_consensus.cc:482] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:02:56.445421 15695 raft_consensus.cc:504] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:56.445689 15695 leader_election.cc:288] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319), 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667)
W20260925 19:02:56.446177 15301 leader_election.cc:341] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319): Not found: Tablet not found: 661e96f0a71b42c8ab642c6fcde34bca
W20260925 19:02:56.446242 15301 leader_election.cc:341] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667): Not found: Tablet not found: 661e96f0a71b42c8ab642c6fcde34bca
I20260925 19:02:56.446327 15301 leader_election.cc:302] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb812bd1639640c0b625ba6b716c537a; no voters: 0c92fabf617e429bb8979d6c1d9f951d, 7618c9e6485b4809be7593519e31ca2b
I20260925 19:02:56.446424 15695 raft_consensus.cc:2726] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:02:56.470211 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:56.470025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:40081)
0925 19:02:56.470131 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:56.470166 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260925 19:02:56.520872 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:56.520666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:36411)
0925 19:02:56.520792 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:56.520832 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260925 19:02:56.655165 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:56.654975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:55607)
0925 19:02:56.655096 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:56.655130 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260925 19:02:57.470806 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:57.470591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:59513)
0925 19:02:57.470723 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:57.470761 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260925 19:02:57.521430 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:57.521243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:32933)
0925 19:02:57.521361 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:57.521391 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":53,"threads_started":1}
I20260925 19:02:57.655851 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:57.655663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:37629)
0925 19:02:57.655779 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:57.655815 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260925 19:02:58.471504 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:58.471249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:44147)
0925 19:02:58.471419 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:58.471458 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260925 19:02:58.522015 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:58.521819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:53945)
0925 19:02:58.521938 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:58.521976 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260925 19:02:58.525691 15704 raft_consensus.cc:482] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:02:58.525796 15704 raft_consensus.cc:504] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:02:58.525972 15704 leader_election.cc:288] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319), 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667)
W20260925 19:02:58.526435 15301 leader_election.cc:341] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667): Not found: Tablet not found: 661e96f0a71b42c8ab642c6fcde34bca
W20260925 19:02:58.526503 15301 leader_election.cc:341] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319): Not found: Tablet not found: 661e96f0a71b42c8ab642c6fcde34bca
I20260925 19:02:58.526520 15301 leader_election.cc:302] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb812bd1639640c0b625ba6b716c537a; no voters: 0c92fabf617e429bb8979d6c1d9f951d, 7618c9e6485b4809be7593519e31ca2b
I20260925 19:02:58.526640 15704 raft_consensus.cc:2726] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:02:58.656571 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:58.656344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:41697)
0925 19:02:58.656478 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:58.656518 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260925 19:02:59.472146 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:59.471931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:36453)
0925 19:02:59.472069 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:59.472100 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260925 19:02:59.522640 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:59.522460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:39455)
0925 19:02:59.522571 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:59.522603 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260925 19:02:59.657187 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:02:59.656990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:35823)
0925 19:02:59.657114 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 19:02:59.657152 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":53,"threads_started":1}
I20260925 19:03:00.472820 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:00.472606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:33009)
0925 19:03:00.472739 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:00.472775 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260925 19:03:00.523259 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:00.523055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:38627)
0925 19:03:00.523182 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:00.523216 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260925 19:03:00.657794 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:00.657611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:47177)
0925 19:03:00.657723 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:00.657754 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260925 19:03:01.473404 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:01.473213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:33645)
0925 19:03:01.473328 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:01.473359 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260925 19:03:01.523972 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:01.523733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:34279)
0925 19:03:01.523892 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:01.523929 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260925 19:03:01.658509 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:01.658319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:33413)
0925 19:03:01.658433 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:01.658469 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260925 19:03:02.474186 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:02.474014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:49951)
0925 19:03:02.474118 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:02.474141 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260925 19:03:02.524637 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:02.524412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:50509)
0925 19:03:02.524561 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:02.524595 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260925 19:03:02.652161 15717 raft_consensus.cc:482] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:03:02.652242 15717 raft_consensus.cc:504] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:03:02.652381 15717 leader_election.cc:288] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319), 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667)
W20260925 19:03:02.652791 15301 leader_election.cc:341] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667): Not found: Tablet not found: 661e96f0a71b42c8ab642c6fcde34bca
W20260925 19:03:02.652848 15301 leader_election.cc:341] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319): Not found: Tablet not found: 661e96f0a71b42c8ab642c6fcde34bca
I20260925 19:03:02.652880 15301 leader_election.cc:302] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb812bd1639640c0b625ba6b716c537a; no voters: 0c92fabf617e429bb8979d6c1d9f951d, 7618c9e6485b4809be7593519e31ca2b
I20260925 19:03:02.652959 15717 raft_consensus.cc:2726] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:03:02.659166 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:02.658950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:52017)
0925 19:03:02.659090 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:02.659118 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260925 19:03:03.474768 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:03.474589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:42903)
0925 19:03:03.474695 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:03.474725 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260925 19:03:03.525207 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:03.525034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:46229)
0925 19:03:03.525139 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:03.525169 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260925 19:03:03.659727 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:03.659536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:55759)
0925 19:03:03.659646 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:03.659675 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260925 19:03:04.475384 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:04.475184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:36577)
0925 19:03:04.475302 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:04.475338 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260925 19:03:04.525795 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:04.525592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:39767)
0925 19:03:04.525724 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:04.525757 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260925 19:03:04.660394 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:04.660215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:37193)
0925 19:03:04.660324 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:04.660356 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260925 19:03:05.475996 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:05.475805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:41545)
0925 19:03:05.475924 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:05.475951 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260925 19:03:05.526473 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:05.526229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:47081)
0925 19:03:05.526398 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:05.526432 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":115,"threads_started":1}
I20260925 19:03:05.661072 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:05.660886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:60351)
0925 19:03:05.661001 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:05.661033 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260925 19:03:06.476603 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:06.476417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:40333)
0925 19:03:06.476526 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:06.476557 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260925 19:03:06.527117 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:06.526932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:54321)
0925 19:03:06.527049 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:06.527078 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260925 19:03:06.661738 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:06.661536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:47915)
0925 19:03:06.661666 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:06.661695 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260925 19:03:07.477288 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:07.477006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:35179)
0925 19:03:07.477127 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:07.477167 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260925 19:03:07.527842 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:07.527641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:37569)
0925 19:03:07.527769 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:07.527799 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":56,"threads_started":1}
I20260925 19:03:07.662343 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:07.662115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:59381)
0925 19:03:07.662223 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:07.662302 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260925 19:03:08.478052 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:08.477865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:60361)
0925 19:03:08.477974 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:08.478002 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
I20260925 19:03:08.528491 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:08.528288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:36881)
0925 19:03:08.528409 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:08.528448 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260925 19:03:08.638818 15736 raft_consensus.cc:482] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:03:08.638911 15736 raft_consensus.cc:504] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c92fabf617e429bb8979d6c1d9f951d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45319 } } peers { permanent_uuid: "7618c9e6485b4809be7593519e31ca2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44667 } } peers { permanent_uuid: "bb812bd1639640c0b625ba6b716c537a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35451 } }
I20260925 19:03:08.639037 15736 leader_election.cc:288] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319), 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667)
W20260925 19:03:08.639485 15301 leader_election.cc:341] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c92fabf617e429bb8979d6c1d9f951d (127.0.98.2:45319): Not found: Tablet not found: 661e96f0a71b42c8ab642c6fcde34bca
W20260925 19:03:08.639575 15301 leader_election.cc:341] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7618c9e6485b4809be7593519e31ca2b (127.0.98.3:44667): Not found: Tablet not found: 661e96f0a71b42c8ab642c6fcde34bca
I20260925 19:03:08.639628 15301 leader_election.cc:302] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb812bd1639640c0b625ba6b716c537a; no voters: 0c92fabf617e429bb8979d6c1d9f951d, 7618c9e6485b4809be7593519e31ca2b
I20260925 19:03:08.639707 15736 raft_consensus.cc:2726] T 661e96f0a71b42c8ab642c6fcde34bca P bb812bd1639640c0b625ba6b716c537a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 19:03:08.654371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 15255
I20260925 19:03:08.659096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 15416
I20260925 19:03:08.662981 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:08.662802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:43233)
0925 19:03:08.662916 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:08.662944 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260925 19:03:08.663105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 15552
2026-09-25T19:03:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18241 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:03:08 dist-test-slave-49bz krb5kdc[15740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:03:08 dist-test-slave-49bz krb5kdc[15740](info): set up 2 sockets
Sep 25 19:03:08 dist-test-slave-49bz krb5kdc[15740](info): commencing operation
krb5kdc: starting...
W20260925 19:03:09.658059 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:03:09 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:03:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:03:09.766793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:03:09.843267 15753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:09.843511 15753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:09.843547 15753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:09.844702 15753 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 19:03:09.844777 15753 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 19:03:09.844798 15753 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 19:03:09.844821 15753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:03:09.844856 15753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:09.844885 15753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:09.844916 15753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:03:09.844947 15753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:03:09.846438 15753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:09.846763 15753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:09.846992 15753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:09.851990 15759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:09.851958 15758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:09.851958 15761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:09.852108 15753 server_base.cc:1034] running on GCE node
I20260925 19:03:09.852438 15753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:09.852825 15753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:09.854012 15753 hybrid_clock.cc:648] HybridClock initialized: now 1790362989853993 us; error 35 us; skew 500 ppm
Sep 25 19:03:09 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:09.856314 15753 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:03:09.856778 15753 webserver.cc:460] Webserver started at http://127.0.98.62:39649/ using document root <none> and password file <none>
I20260925 19:03:09.856997 15753 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:09.857064 15753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:09.857218 15753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:03:09.858301 15753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "245c2fc6202348f3b5ebce7a7debb3c5"
format_stamp: "Formatted at 2026-09-25 19:03:09 on dist-test-slave-49bz"
server_key: "470cab1b7dbfcfdcee233cff9316a4f2"
server_key_iv: "e31c5c58e2d4c17545f87d16bb900e14"
server_key_version: "encryptionkey@0"
I20260925 19:03:09.858677 15753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "245c2fc6202348f3b5ebce7a7debb3c5"
format_stamp: "Formatted at 2026-09-25 19:03:09 on dist-test-slave-49bz"
server_key: "470cab1b7dbfcfdcee233cff9316a4f2"
server_key_iv: "e31c5c58e2d4c17545f87d16bb900e14"
server_key_version: "encryptionkey@0"
I20260925 19:03:09.860262 15753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:03:09.861418 15768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:09.861601 15753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:03:09.861719 15753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "245c2fc6202348f3b5ebce7a7debb3c5"
format_stamp: "Formatted at 2026-09-25 19:03:09 on dist-test-slave-49bz"
server_key: "470cab1b7dbfcfdcee233cff9316a4f2"
server_key_iv: "e31c5c58e2d4c17545f87d16bb900e14"
server_key_version: "encryptionkey@0"
I20260925 19:03:09.861788 15753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:09.870885 15753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:09.872473 15753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:09.872620 15753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:09.876261 15753 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37199
I20260925 19:03:09.876289 15820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37199 every 8 connection(s)
I20260925 19:03:09.876637 15753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:03:09.877418 15821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:03:09.879566 15821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Bootstrap starting.
I20260925 19:03:09.880087 15821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Neither blocks nor log segments found. Creating new log.
I20260925 19:03:09.880324 15821 log.cc:824] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:09.880789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 15753
I20260925 19:03:09.880888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:03:09.881047 15821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: No bootstrap required, opened a new log
I20260925 19:03:09.881112 392 external_mini_cluster.cc:1428] Setting key 6d2681315795e5f6c40916d5b93c8ed8
I20260925 19:03:09.883293 15821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:09.883414 15821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:03:09.883435 15821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245c2fc6202348f3b5ebce7a7debb3c5, State: Initialized, Role: FOLLOWER
I20260925 19:03:09.883558 15821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:09.883649 15821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:03:09.883677 15821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:03:09.883752 15821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:03:09.884243 15821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:09.884316 15821 leader_election.cc:302] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 245c2fc6202348f3b5ebce7a7debb3c5; no voters:
I20260925 19:03:09.884503 15821 leader_election.cc:288] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:03:09.884524 15826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:03:09.884629 15826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 1 LEADER]: Becoming Leader. State: Replica: 245c2fc6202348f3b5ebce7a7debb3c5, State: Running, Role: LEADER
I20260925 19:03:09.884743 15821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:03:09.884748 15826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:09.885049 15826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 245c2fc6202348f3b5ebce7a7debb3c5. Latest consensus state: current_term: 1 leader_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } } }
I20260925 19:03:09.885504 15826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:03:09.885407 15827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } } }
I20260925 19:03:09.885953 15827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:03:09.886109 15835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:03:09.886706 15835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 19:03:09 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:03:09.890636 15835 catalog_manager.cc:1347] Generated new cluster ID: 1edcd605ca0741168f6fdb0c303ad99d
I20260925 19:03:09.890686 15835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:03:09.892700 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:09.881574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.0.98.62:37199)
0925 19:03:09.881752 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:09.881761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:09.881791 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:03:09.882286 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:09.882294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:09.882349 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:09.882449 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:09.883221 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:09.884610 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:09.885932 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:09.886653 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:09.886778 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:09.889276 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:09.889279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:09.889279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:09.889291 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:09.891123 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:09.891470 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:09.891473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:09.891473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:09.891573 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:09.891758 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:09.891759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:09.891759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:09.892049 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:09.892178 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:09.892274 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 19:03:09.892398 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260925 19:03:09.912758 15835 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:03:09.913486 15835 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:03:09.945243 15835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Generated new TSK 0
I20260925 19:03:09.945447 15835 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:03:09.975098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:03:10.053900 15847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:10.054095 15847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:10.054118 15847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:10.055269 15847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:10.055312 15847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:10.055338 15847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:03:10.056762 15847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:10.057008 15847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:10.057232 15847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:10.062026 15853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:10.062034 15852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:10.062047 15855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:10.062357 15847 server_base.cc:1034] running on GCE node
I20260925 19:03:10.062615 15847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:10.062849 15847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:10.063994 15847 hybrid_clock.cc:648] HybridClock initialized: now 1790362990063982 us; error 28 us; skew 500 ppm
Sep 25 19:03:10 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:10.066653 15847 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:03:10.067037 15847 webserver.cc:460] Webserver started at http://127.0.98.1:34435/ using document root <none> and password file <none>
I20260925 19:03:10.067234 15847 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:10.067276 15847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:10.067364 15847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:03:10.068349 15847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "c8bfce2105004a0c8c68de914257f81a"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "76dcebb7723d42d2ccd560298a48023e"
server_key_iv: "a0b273a860d249f51b4362c5b62f25bb"
server_key_version: "encryptionkey@0"
I20260925 19:03:10.068666 15847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c8bfce2105004a0c8c68de914257f81a"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "76dcebb7723d42d2ccd560298a48023e"
server_key_iv: "a0b273a860d249f51b4362c5b62f25bb"
server_key_version: "encryptionkey@0"
I20260925 19:03:10.069895 15847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:03:10.070649 15862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:10.070813 15847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:10.070917 15847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "c8bfce2105004a0c8c68de914257f81a"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "76dcebb7723d42d2ccd560298a48023e"
server_key_iv: "a0b273a860d249f51b4362c5b62f25bb"
server_key_version: "encryptionkey@0"
I20260925 19:03:10.070991 15847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:10.090904 15847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:10.092742 15847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:10.092907 15847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:10.093134 15847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:03:10.093441 15847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:03:10.093500 15847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:10.093533 15847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:03:10.093559 15847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:10.099401 15847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42249
I20260925 19:03:10.099454 15975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42249 every 8 connection(s)
I20260925 19:03:10.099776 15847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:03:10 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:03:10.108568 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.100545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44443 (local address 127.0.98.62:37199)
0925 19:03:10.100750 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:10.100752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:10.100787 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:03:10.101274 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:10.101276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:10.101286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:10.101351 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:10.102343 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.102877 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.103830 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.104014 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.104061 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.106048 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:10.106049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:10.106050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:10.106056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:10.107508 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:10.107846 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:10.107847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:10.107847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:10.107895 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:10.108088 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:10.108089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:10.108089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:10.108180 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:10.108226 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:10.108438 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 19:03:10.108504 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260925 19:03:10.108700 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.100517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:44443)
0925 19:03:10.100663 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:10.100706 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:10.101186 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:10.101406 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:10.101409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:10.101994 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:03:10.102199 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.102204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.102964 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.102965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.103759 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.103761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.104082 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.104083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.104186 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.104635 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:03:10.104643 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:03:10.105967 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:03:10.107572 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:10.107577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:10.107579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:10.107788 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:10.107949 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:10.107949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:10.107949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:10.108039 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:10.108231 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:03:10.108235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:03:10.108340 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 19:03:10.108417 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:03:10.108523 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260925 19:03:10.109200 15976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37199
I20260925 19:03:10.109297 15976 heartbeater.cc:464] Registering TS with master...
I20260925 19:03:10.109526 15976 heartbeater.cc:511] Master 127.0.98.62:37199 requested a full tablet report, sending...
I20260925 19:03:10.109835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 15847
I20260925 19:03:10.109954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:03:10.110136 15785 ts_manager.cc:194] Registered new tserver with Master: c8bfce2105004a0c8c68de914257f81a (127.0.98.1:42249)
I20260925 19:03:10.110183 392 external_mini_cluster.cc:1428] Setting key 5cf6c19d581768f8e6ff4a03a0622814
I20260925 19:03:10.110826 15785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44443
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:03:10.133342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:03:10.218205 15981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:10.218431 15981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:10.218454 15981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:10.219488 15981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:10.219521 15981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:10.219547 15981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:03:10.220908 15981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:10.221091 15981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:10.221267 15981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:10.226318 15987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:10.226289 15986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:10.226306 15989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:10.226434 15981 server_base.cc:1034] running on GCE node
I20260925 19:03:10.226774 15981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:10.227021 15981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:10.228174 15981 hybrid_clock.cc:648] HybridClock initialized: now 1790362990228147 us; error 38 us; skew 500 ppm
Sep 25 19:03:10 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:10.230518 15981 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:03:10.230916 15981 webserver.cc:460] Webserver started at http://127.0.98.2:37303/ using document root <none> and password file <none>
I20260925 19:03:10.231096 15981 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:10.231135 15981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:10.231252 15981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:03:10.232185 15981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "0a34a864ea3f4030a6b3b30499ec662a"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "c88bb26e8591f0953a8e4f6c583ead16"
server_key_iv: "68b87308a5b86eceb75d7bb858bafb6a"
server_key_version: "encryptionkey@0"
I20260925 19:03:10.232457 15981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0a34a864ea3f4030a6b3b30499ec662a"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "c88bb26e8591f0953a8e4f6c583ead16"
server_key_iv: "68b87308a5b86eceb75d7bb858bafb6a"
server_key_version: "encryptionkey@0"
I20260925 19:03:10.233721 15981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:03:10.234424 15996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:10.234611 15981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:03:10.234694 15981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "0a34a864ea3f4030a6b3b30499ec662a"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "c88bb26e8591f0953a8e4f6c583ead16"
server_key_iv: "68b87308a5b86eceb75d7bb858bafb6a"
server_key_version: "encryptionkey@0"
I20260925 19:03:10.234742 15981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:10.245903 15981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:10.248172 15981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:10.248281 15981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:10.248443 15981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:03:10.248747 15981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:03:10.248804 15981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:10.248845 15981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:03:10.248873 15981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:10.254642 15981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40633
I20260925 19:03:10.254681 16109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40633 every 8 connection(s)
I20260925 19:03:10.254997 15981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:03:10.258433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 15981
I20260925 19:03:10.258548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:03:10.258740 392 external_mini_cluster.cc:1428] Setting key e2a19844afbbdabf10a465467214873c
Sep 25 19:03:10 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:03:10.264568 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.255714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37923 (local address 127.0.98.62:37199)
0925 19:03:10.255778 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:10.255780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:10.255957 (+ 177us) server_negotiation.cc:407] Connection header received
0925 19:03:10.256409 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:10.256410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:10.256419 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:10.256474 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:10.257365 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.257859 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.259185 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.259435 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.259519 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.261555 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:10.261556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:10.261557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:10.261565 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:10.263223 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:10.263587 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:10.263589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:10.263589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:10.263657 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:10.263899 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:10.263900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:10.263901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:10.264052 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:10.264128 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:10.264371 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 19:03:10.264461 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:03:10.264665 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.255691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:37923)
0925 19:03:10.255832 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:10.255875 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:10.256333 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:10.256529 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:10.256532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:10.257081 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:03:10.257271 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.257275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.257941 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.257942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.259120 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.259122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.259524 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.259525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.259667 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.260180 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:03:10.260188 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:03:10.261483 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:03:10.263304 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:10.263309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:10.263311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:10.263512 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:10.263734 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:10.263735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:10.263735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:10.263830 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:10.264121 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:03:10.264125 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:03:10.264242 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 19:03:10.264328 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:03:10.264440 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260925 19:03:10.265122 16110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37199
I20260925 19:03:10.265208 16110 heartbeater.cc:464] Registering TS with master...
I20260925 19:03:10.265384 16110 heartbeater.cc:511] Master 127.0.98.62:37199 requested a full tablet report, sending...
I20260925 19:03:10.265753 15785 ts_manager.cc:194] Registered new tserver with Master: 0a34a864ea3f4030a6b3b30499ec662a (127.0.98.2:40633)
I20260925 19:03:10.266078 15785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37923
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:03:10.283389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:03:10.362423 16115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:10.362668 16115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:10.362708 16115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:10.363842 16115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:10.363916 16115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:10.363960 16115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:03:10.365381 16115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:10.365633 16115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:10.365868 16115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:10.370656 16121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:10.370642 16123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:10.370854 16115 server_base.cc:1034] running on GCE node
W20260925 19:03:10.370656 16120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:10.371099 16115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:10.371281 16115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:10.372449 16115 hybrid_clock.cc:648] HybridClock initialized: now 1790362990372442 us; error 33 us; skew 500 ppm
Sep 25 19:03:10 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:10.375119 16115 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:03:10.375478 16115 webserver.cc:460] Webserver started at http://127.0.98.3:46241/ using document root <none> and password file <none>
I20260925 19:03:10.375650 16115 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:10.375689 16115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:10.375770 16115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:03:10.376715 16115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "d293cc736c514dcf87e50a37157093fb"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "884a430778213a3eca44dbacb4f4c1fb"
server_key_iv: "89d337d13e9ba47d0c6cf7ce34d1058d"
server_key_version: "encryptionkey@0"
I20260925 19:03:10.377002 16115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d293cc736c514dcf87e50a37157093fb"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "884a430778213a3eca44dbacb4f4c1fb"
server_key_iv: "89d337d13e9ba47d0c6cf7ce34d1058d"
server_key_version: "encryptionkey@0"
I20260925 19:03:10.378222 16115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:03:10.378930 16130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:10.379110 16115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:03:10.379195 16115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "d293cc736c514dcf87e50a37157093fb"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "884a430778213a3eca44dbacb4f4c1fb"
server_key_iv: "89d337d13e9ba47d0c6cf7ce34d1058d"
server_key_version: "encryptionkey@0"
I20260925 19:03:10.379266 16115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:10.390008 16115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:10.392087 16115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:10.392227 16115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:10.392463 16115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:03:10.392760 16115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:03:10.392812 16115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:10.392853 16115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:03:10.392881 16115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:10.398460 16115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43569
I20260925 19:03:10.398568 16243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43569 every 8 connection(s)
I20260925 19:03:10.398854 16115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:03:10 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790362990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:03:10.407559 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.399611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45667 (local address 127.0.98.62:37199)
0925 19:03:10.399670 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:10.399672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:10.399852 (+ 180us) server_negotiation.cc:407] Connection header received
0925 19:03:10.400321 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:10.400322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:10.400331 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:10.400388 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:10.401275 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.401774 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.402837 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.403021 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.403079 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.405040 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:10.405041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:10.405042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:10.405048 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:10.406488 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:10.406828 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:10.406829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:10.406829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:10.406877 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:10.407071 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:10.407071 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:10.407072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:10.407167 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:10.407222 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:10.407441 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 19:03:10.407496 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:03:10.407723 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.399590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:45667)
0925 19:03:10.399732 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:10.399773 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:10.400241 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:10.400440 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:10.400443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:10.401007 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:03:10.401200 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.401205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.401852 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.401853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.402760 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.402763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.403110 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.403111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.403206 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.403644 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:03:10.403651 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:03:10.404964 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:03:10.406555 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:10.406560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:10.406562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:10.406767 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:10.406932 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:10.406933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:10.406933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:10.407021 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:10.407227 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:03:10.407231 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:03:10.407338 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 19:03:10.407416 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 19:03:10.407524 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260925 19:03:10.408043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 16115
I20260925 19:03:10.408175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:03:10.408254 16244 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37199
I20260925 19:03:10.408347 16244 heartbeater.cc:464] Registering TS with master...
I20260925 19:03:10.408392 392 external_mini_cluster.cc:1428] Setting key a260692d520b1014e06ef1869edeebd1
I20260925 19:03:10.408566 16244 heartbeater.cc:511] Master 127.0.98.62:37199 requested a full tablet report, sending...
I20260925 19:03:10.408946 15785 ts_manager.cc:194] Registered new tserver with Master: d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569)
I20260925 19:03:10.409328 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:03:10.409425 15785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45667
I20260925 19:03:10.416366 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.410647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.0.98.62:37199)
0925 19:03:10.410682 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:10.410684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:10.410802 (+ 118us) server_negotiation.cc:407] Connection header received
0925 19:03:10.410875 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:10.410876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:10.410885 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:10.410927 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:10.411468 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.411929 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.412660 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.412846 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.412924 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.413998 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:10.413999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:10.413999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:10.414006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:10.415211 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:10.415576 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:10.415577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:10.415577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:10.415635 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:10.415848 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:10.415848 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:10.415849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:10.415993 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:10.416098 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:10.416196 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 19:03:10.416271 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:03:10.417879 15785 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:03:10.419600 15785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:03:10.425940 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.421831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43569 (local address 127.0.0.1:50540)
0925 19:03:10.422010 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:10.422046 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:10.422121 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:10.422355 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:10.422358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:10.422360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:03:10.422421 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.422422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.423748 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.423749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.424811 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.424813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.425640 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.425641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.425724 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.425737 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:03:10.425812 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:03:10.425861 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":52,"thread_start_us":81,"threads_started":1}
I20260925 19:03:10.425971 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.421907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.0.98.3:43569)
0925 19:03:10.422198 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:10.422202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:10.422211 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:03:10.422242 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:10.422244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:10.422281 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:10.422354 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:03:10.422527 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.423642 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.424899 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.425534 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.425639 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.425695 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:10.425837 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 19:03:10.425878 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":74,"threads_started":1}
I20260925 19:03:10.426405 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.422215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.0.98.2:40633)
0925 19:03:10.422619 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:10.422624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:10.422633 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:03:10.422671 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:10.422673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:10.422688 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:10.422762 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:03:10.422944 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.424080 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.425312 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.425924 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.425985 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.426055 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:10.426303 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 19:03:10.426348 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":126,"threads_started":1}
I20260925 19:03:10.426435 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.421873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42249 (local address 127.0.0.1:39930)
0925 19:03:10.422049 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:10.422060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:10.422121 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:10.422716 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:10.422717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:10.422719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:03:10.422769 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.422770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.423828 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.423829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.425062 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.425065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.425950 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.425951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.426016 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.426021 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:03:10.426057 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 19:03:10.426113 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":26,"thread_start_us":51,"threads_started":1}
I20260925 19:03:10.426563 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.422047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.0.1:49098)
0925 19:03:10.422349 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:10.422358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:10.422403 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:10.422837 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:10.422838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:10.422840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:03:10.422876 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.422877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.424191 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.424192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.425238 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.425240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.426088 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.426089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.426149 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.426152 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:03:10.426195 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 19:03:10.426229 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":41,"threads_started":1}
I20260925 19:03:10.426811 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.422003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.0.98.1:42249)
0925 19:03:10.422533 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:10.422538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:10.422548 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:03:10.422598 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:10.422600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:10.422616 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:10.422929 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:03:10.423007 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.423667 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.425151 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.425858 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.426599 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.426663 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:10.426688 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 19:03:10.426727 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":80,"threads_started":1}
I20260925 19:03:10.426824 16178 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef5cc287e014469b86d908b5aef1486 (DEFAULT_TABLE table=test-table [id=279be61c0a9b466999aa2718751da7fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:03:10.427181 16178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef5cc287e014469b86d908b5aef1486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:03:10.427187 16044 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef5cc287e014469b86d908b5aef1486 (DEFAULT_TABLE table=test-table [id=279be61c0a9b466999aa2718751da7fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:03:10.427423 16044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef5cc287e014469b86d908b5aef1486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:03:10.429533 16258 tablet_bootstrap.cc:492] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: Bootstrap starting.
I20260925 19:03:10.430030 15910 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef5cc287e014469b86d908b5aef1486 (DEFAULT_TABLE table=test-table [id=279be61c0a9b466999aa2718751da7fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:03:10.430187 16258 tablet_bootstrap.cc:654] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: Neither blocks nor log segments found. Creating new log.
I20260925 19:03:10.430375 15910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef5cc287e014469b86d908b5aef1486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:03:10.430791 16258 log.cc:824] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:10.431514 16258 tablet_bootstrap.cc:492] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: No bootstrap required, opened a new log
I20260925 19:03:10.431595 16258 ts_tablet_manager.cc:1397] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:03:10.433388 16259 tablet_bootstrap.cc:492] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: Bootstrap starting.
I20260925 19:03:10.433605 16258 raft_consensus.cc:348] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:10.433770 16258 raft_consensus.cc:374] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:03:10.433794 16258 raft_consensus.cc:729] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d293cc736c514dcf87e50a37157093fb, State: Initialized, Role: FOLLOWER
I20260925 19:03:10.433945 16258 consensus_queue.cc:260] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:10.434080 16259 tablet_bootstrap.cc:654] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: Neither blocks nor log segments found. Creating new log.
I20260925 19:03:10.434217 16258 ts_tablet_manager.cc:1428] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:03:10.434295 16244 heartbeater.cc:503] Master 127.0.98.62:37199 was elected leader, sending a full tablet report...
I20260925 19:03:10.434417 16261 tablet_bootstrap.cc:492] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Bootstrap starting.
I20260925 19:03:10.434720 16259 log.cc:824] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:10.435065 16261 tablet_bootstrap.cc:654] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Neither blocks nor log segments found. Creating new log.
I20260925 19:03:10.435410 16259 tablet_bootstrap.cc:492] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: No bootstrap required, opened a new log
I20260925 19:03:10.435489 16259 ts_tablet_manager.cc:1397] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:03:10.435652 16261 log.cc:824] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:10.436275 16261 tablet_bootstrap.cc:492] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: No bootstrap required, opened a new log
I20260925 19:03:10.436324 16261 ts_tablet_manager.cc:1397] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:03:10.436992 16259 raft_consensus.cc:348] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:10.437192 16259 raft_consensus.cc:374] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:03:10.437239 16259 raft_consensus.cc:729] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a34a864ea3f4030a6b3b30499ec662a, State: Initialized, Role: FOLLOWER
I20260925 19:03:10.437398 16259 consensus_queue.cc:260] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:10.437669 16259 ts_tablet_manager.cc:1428] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:03:10.437745 16110 heartbeater.cc:503] Master 127.0.98.62:37199 was elected leader, sending a full tablet report...
I20260925 19:03:10.438051 16261 raft_consensus.cc:348] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:10.438244 16261 raft_consensus.cc:374] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:03:10.438297 16261 raft_consensus.cc:729] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8bfce2105004a0c8c68de914257f81a, State: Initialized, Role: FOLLOWER
I20260925 19:03:10.438406 16261 consensus_queue.cc:260] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:10.438678 16261 ts_tablet_manager.cc:1428] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 19:03:10.438766 15976 heartbeater.cc:503] Master 127.0.98.62:37199 was elected leader, sending a full tablet report...
I20260925 19:03:10.455711 16266 raft_consensus.cc:482] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:03:10.455814 16266 raft_consensus.cc:504] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:10.456099 16266 leader_election.cc:288] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569), 0a34a864ea3f4030a6b3b30499ec662a (127.0.98.2:40633)
I20260925 19:03:10.459754 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.456262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43569 (local address 127.0.98.1:59905)
0925 19:03:10.456433 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:10.456445 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:10.456506 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:10.456656 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:10.456657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:10.456659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:03:10.456726 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.456728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.457551 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.457552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.458925 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.458928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.459492 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.459492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.459546 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.459549 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:03:10.459629 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 19:03:10.459670 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":78,"thread_start_us":96,"threads_started":1}
I20260925 19:03:10.459748 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.456333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59905 (local address 127.0.98.3:43569)
0925 19:03:10.456433 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:10.456435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:10.456514 (+ 79us) server_negotiation.cc:407] Connection header received
0925 19:03:10.456570 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:10.456571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:10.456584 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:10.456638 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:03:10.456793 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.457477 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.459031 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.459417 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.459482 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.459491 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:10.459637 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 19:03:10.459683 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 19:03:10.460140 16198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef5cc287e014469b86d908b5aef1486" candidate_uuid: "c8bfce2105004a0c8c68de914257f81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d293cc736c514dcf87e50a37157093fb" is_pre_election: true
I20260925 19:03:10.460318 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.456262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.98.1:52725)
0925 19:03:10.456324 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:10.456337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:10.456379 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:10.456538 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:10.456539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:10.456540 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:03:10.456615 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.456617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.457561 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.457562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.458925 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:10.458928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.460093 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.460094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:10.460161 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.460164 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:03:10.460207 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 19:03:10.460237 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":59}
I20260925 19:03:10.460326 16198 raft_consensus.cc:2454] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8bfce2105004a0c8c68de914257f81a in term 0.
I20260925 19:03:10.460348 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.456343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52725 (local address 127.0.98.2:40633)
0925 19:03:10.456383 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:10.456386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:10.456428 (+ 42us) server_negotiation.cc:407] Connection header received
0925 19:03:10.456458 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:10.456459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:10.456470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:10.456524 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:03:10.456700 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.457456 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.459520 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.460005 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.460083 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.460093 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:10.460238 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 19:03:10.460282 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:03:10.460533 15863 leader_election.cc:302] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8bfce2105004a0c8c68de914257f81a, d293cc736c514dcf87e50a37157093fb; no voters:
I20260925 19:03:10.460702 16064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef5cc287e014469b86d908b5aef1486" candidate_uuid: "c8bfce2105004a0c8c68de914257f81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a34a864ea3f4030a6b3b30499ec662a" is_pre_election: true
I20260925 19:03:10.460728 16266 raft_consensus.cc:2781] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:03:10.460785 16266 raft_consensus.cc:482] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:03:10.460801 16266 raft_consensus.cc:3037] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:03:10.460837 16064 raft_consensus.cc:2454] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8bfce2105004a0c8c68de914257f81a in term 0.
I20260925 19:03:10.461534 16266 raft_consensus.cc:504] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:10.461680 16266 leader_election.cc:288] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [CANDIDATE]: Term 1 election: Requested vote from peers d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569), 0a34a864ea3f4030a6b3b30499ec662a (127.0.98.2:40633)
I20260925 19:03:10.461848 16198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef5cc287e014469b86d908b5aef1486" candidate_uuid: "c8bfce2105004a0c8c68de914257f81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d293cc736c514dcf87e50a37157093fb"
I20260925 19:03:10.461839 16064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef5cc287e014469b86d908b5aef1486" candidate_uuid: "c8bfce2105004a0c8c68de914257f81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a34a864ea3f4030a6b3b30499ec662a"
I20260925 19:03:10.461917 16198 raft_consensus.cc:3037] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:03:10.461916 16064 raft_consensus.cc:3037] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:03:10.462845 16198 raft_consensus.cc:2454] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8bfce2105004a0c8c68de914257f81a in term 1.
I20260925 19:03:10.462845 16064 raft_consensus.cc:2454] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8bfce2105004a0c8c68de914257f81a in term 1.
I20260925 19:03:10.463023 15863 leader_election.cc:302] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8bfce2105004a0c8c68de914257f81a, d293cc736c514dcf87e50a37157093fb; no voters:
I20260925 19:03:10.463155 16266 raft_consensus.cc:2781] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:03:10.463325 16266 raft_consensus.cc:686] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 1 LEADER]: Becoming Leader. State: Replica: c8bfce2105004a0c8c68de914257f81a, State: Running, Role: LEADER
I20260925 19:03:10.463425 16266 consensus_queue.cc:237] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:10.464206 15785 catalog_manager.cc:5495] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a reported cstate change: term changed from 0 to 1, leader changed from <none> to c8bfce2105004a0c8c68de914257f81a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c8bfce2105004a0c8c68de914257f81a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } health_report { overall_health: UNKNOWN } } }
I20260925 19:03:10.470068 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.468055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.0.98.1:42249)
0925 19:03:10.468089 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:10.468091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:10.468101 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:03:10.468176 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:10.468176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:10.468191 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:10.468239 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:03:10.468312 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.468751 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.469410 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:10.469573 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:10.469619 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:10.469801 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:03:10.469883 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:03:10.469940 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:10.470006 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 19:03:10.470022 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:03:10.471608 16198 raft_consensus.cc:1261] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb [term 1 FOLLOWER]: Refusing update from remote peer c8bfce2105004a0c8c68de914257f81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:03:10.471655 16064 raft_consensus.cc:1261] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Refusing update from remote peer c8bfce2105004a0c8c68de914257f81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:03:10.471942 16268 consensus_queue.cc:1035] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:03:10.472064 16266 consensus_queue.cc:1035] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:03:10.474218 16271 mvcc.cc:205] Tried to move back new op lower bound from 7333326808969494528 to 7333326808938770432. Current Snapshot: MvccSnapshot[applied={T|T < 7333326808969494528}]
I20260925 19:03:10.475162 16273 mvcc.cc:205] Tried to move back new op lower bound from 7333326808969494528 to 7333326808938770432. Current Snapshot: MvccSnapshot[applied={T|T < 7333326808969494528}]
I20260925 19:03:10.476460 15784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:03:10.476580 15784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:03:10.477295 15784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Sending DeleteTablet for 3 replicas of tablet 2ef5cc287e014469b86d908b5aef1486
I20260925 19:03:10.477639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 15847
I20260925 19:03:10.477631 16178 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef5cc287e014469b86d908b5aef1486 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:03:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50540
I20260925 19:03:10.477641 16044 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef5cc287e014469b86d908b5aef1486 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:03:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49098
I20260925 19:03:10.479055 16281 tablet_replica.cc:330] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: stopping tablet replica
I20260925 19:03:10.479137 16281 raft_consensus.cc:2229] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:03:10.479207 16281 raft_consensus.cc:2258] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:03:10.479349 16280 tablet_replica.cc:330] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: stopping tablet replica
I20260925 19:03:10.479434 16280 raft_consensus.cc:2229] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:03:10.479518 16280 raft_consensus.cc:2258] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:03:10.479686 16281 ts_tablet_manager.cc:1905] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:03:10.479879 16280 ts_tablet_manager.cc:1905] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:03:10.481376 16280 ts_tablet_manager.cc:1918] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:03:10.481441 16280 log.cc:1196] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/2ef5cc287e014469b86d908b5aef1486
I20260925 19:03:10.481604 16281 ts_tablet_manager.cc:1918] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:03:10.481681 16281 log.cc:1196] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/2ef5cc287e014469b86d908b5aef1486
I20260925 19:03:10.481814 16280 ts_tablet_manager.cc:1939] T 2ef5cc287e014469b86d908b5aef1486 P d293cc736c514dcf87e50a37157093fb: Deleting consensus metadata
I20260925 19:03:10.481980 16281 ts_tablet_manager.cc:1939] T 2ef5cc287e014469b86d908b5aef1486 P 0a34a864ea3f4030a6b3b30499ec662a: Deleting consensus metadata
I20260925 19:03:10.482218 15769 catalog_manager.cc:4841] TS d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569): tablet 2ef5cc287e014469b86d908b5aef1486 (table test-table [id=279be61c0a9b466999aa2718751da7fe]) successfully deleted
I20260925 19:03:10.482326 15770 catalog_manager.cc:4841] TS 0a34a864ea3f4030a6b3b30499ec662a (127.0.98.2:40633): tablet 2ef5cc287e014469b86d908b5aef1486 (table test-table [id=279be61c0a9b466999aa2718751da7fe]) successfully deleted
W20260925 19:03:10.483501 15770 connection.cc:535] client connection to 127.0.98.1:42249 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 19:03:10.483582 15770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:03:10.483748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 15981
I20260925 19:03:10.484005 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:10.483898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42249 (local address 127.0.0.1:39940)
0925 19:03:10.483942 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:10.483971 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 19:03:10.484094 15770 catalog_manager.cc:4584] TS c8bfce2105004a0c8c68de914257f81a (127.0.98.1:42249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ef5cc287e014469b86d908b5aef1486: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42249: connect: Connection refused (error 111)
I20260925 19:03:10.489423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 16115
I20260925 19:03:10.493832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 15753
I20260925 19:03:30.499315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37199
--webserver_interface=127.0.98.62
--webserver_port=39649
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:03:30.580076 16283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:30.580271 16283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:30.580293 16283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:30.581540 16283 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 19:03:30.581584 16283 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 19:03:30.581598 16283 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 19:03:30.581609 16283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:03:30.581622 16283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:30.581674 16283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:30.581692 16283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:03:30.581715 16283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:03:30.583330 16283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:30.583581 16283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:30.583815 16283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:30.592239 16290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:30.592458 16283 server_base.cc:1034] running on GCE node
W20260925 19:03:30.592227 16292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:30.592259 16289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:30.592849 16283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:30.593145 16283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:30.594311 16283 hybrid_clock.cc:648] HybridClock initialized: now 1790363010594291 us; error 27 us; skew 500 ppm
Sep 25 19:03:30 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:30.596755 16283 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:03:30.597182 16283 webserver.cc:460] Webserver started at http://127.0.98.62:39649/ using document root <none> and password file <none>
I20260925 19:03:30.597414 16283 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:30.597491 16283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:30.598912 16283 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:03:30.599545 16299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:30.599705 16283 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:03:30.599810 16283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "245c2fc6202348f3b5ebce7a7debb3c5"
format_stamp: "Formatted at 2026-09-25 19:03:09 on dist-test-slave-49bz"
server_key: "470cab1b7dbfcfdcee233cff9316a4f2"
server_key_iv: "e31c5c58e2d4c17545f87d16bb900e14"
server_key_version: "encryptionkey@0"
I20260925 19:03:30.600215 16283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:30.617225 16283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:30.619169 16283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:30.619325 16283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:30.623256 16283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37199
I20260925 19:03:30.623293 16351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37199 every 8 connection(s)
I20260925 19:03:30.623692 16283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:03:30.624202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 16283
I20260925 19:03:30.624547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42249
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37199
--webserver_port=34435
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:03:30.625916 16352 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:03:30.626638 16352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Bootstrap starting.
I20260925 19:03:30.634979 16352 log.cc:824] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:30.636909 16352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:03:30.637198 16352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Bootstrap complete.
I20260925 19:03:30.639549 16352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:30.639765 16352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245c2fc6202348f3b5ebce7a7debb3c5, State: Initialized, Role: FOLLOWER
I20260925 19:03:30.639936 16352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:30.640012 16352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:03:30.640084 16352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:03:30.640129 16352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:03:30.641093 16352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:30.641199 16352 leader_election.cc:302] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 245c2fc6202348f3b5ebce7a7debb3c5; no voters:
I20260925 19:03:30.641384 16352 leader_election.cc:288] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 19:03:30.641418 16356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:03:30.641592 16356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 2 LEADER]: Becoming Leader. State: Replica: 245c2fc6202348f3b5ebce7a7debb3c5, State: Running, Role: LEADER
I20260925 19:03:30.641654 16352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:03:30.641732 16356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:30.642014 16356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 245c2fc6202348f3b5ebce7a7debb3c5. Latest consensus state: current_term: 2 leader_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } } }
I20260925 19:03:30.642019 16357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } } }
I20260925 19:03:30.642284 16357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:03:30.642624 16356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:03:30.642678 16365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:03:30.643680 16365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:03:30.643782 16365 catalog_manager.cc:1259] Loaded cluster ID: 1edcd605ca0741168f6fdb0c303ad99d
I20260925 19:03:30.643812 16365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:03:30.644961 16365 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:03:30.645128 16365 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Loaded TSK: 0
I20260925 19:03:30.663515 16365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Generated new TSK 1
I20260925 19:03:30.663622 16365 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:03:30.710705 16354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:30.710942 16354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:30.710970 16354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:30.712116 16354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:30.712154 16354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:30.712179 16354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:03:30.713552 16354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34435
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:30.713768 16354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:30.713999 16354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:30.718731 16380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:30.718735 16379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:30.718955 16354 server_base.cc:1034] running on GCE node
W20260925 19:03:30.719066 16382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:30.719269 16354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:30.719560 16354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:30.720737 16354 hybrid_clock.cc:648] HybridClock initialized: now 1790363010720724 us; error 29 us; skew 500 ppm
Sep 25 19:03:30 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:30.723649 16354 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:03:30.724050 16354 webserver.cc:460] Webserver started at http://127.0.98.1:34435/ using document root <none> and password file <none>
I20260925 19:03:30.724280 16354 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:30.724363 16354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:30.725811 16354 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:03:30.726637 16389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:30.726815 16354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:03:30.726917 16354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "c8bfce2105004a0c8c68de914257f81a"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "76dcebb7723d42d2ccd560298a48023e"
server_key_iv: "a0b273a860d249f51b4362c5b62f25bb"
server_key_version: "encryptionkey@0"
I20260925 19:03:30.727270 16354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:30.747782 16354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:30.750056 16354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:30.750217 16354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:30.750494 16354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:03:30.750965 16396 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:03:30.752780 16354 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:03:30.752841 16354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:03:30.752883 16354 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:03:30.753482 16354 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:03:30.753533 16354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:03:30.753587 16396 tablet_bootstrap.cc:492] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Bootstrap starting.
I20260925 19:03:30.760672 16354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42249
I20260925 19:03:30.760751 16503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42249 every 8 connection(s)
I20260925 19:03:30.761086 16354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:03:30.764122 16396 log.cc:824] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:30.766894 16396 tablet_bootstrap.cc:492] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:03:30.767189 16396 tablet_bootstrap.cc:492] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Bootstrap complete.
I20260925 19:03:30.768374 16396 ts_tablet_manager.cc:1397] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
Sep 25 19:03:30 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:03:30.769740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 16354
I20260925 19:03:30.769655 16396 raft_consensus.cc:348] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:30.770031 16396 raft_consensus.cc:729] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8bfce2105004a0c8c68de914257f81a, State: Initialized, Role: FOLLOWER
I20260925 19:03:30.770057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40633
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37199
--webserver_port=37303
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:03:30.770370 16396 consensus_queue.cc:260] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } }
I20260925 19:03:30.770680 16396 ts_tablet_manager.cc:1428] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260925 19:03:30.774343 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:30.761853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41175 (local address 127.0.98.62:37199)
0925 19:03:30.764357 (+ 2504us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:30.764367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:30.764413 (+ 46us) server_negotiation.cc:407] Connection header received
0925 19:03:30.764558 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:30.764565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:30.764631 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:30.764785 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:30.765369 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.766348 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.767500 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.767878 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.767988 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:30.770400 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:30.770404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:30.770405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:30.770416 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:30.772266 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:30.772805 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:30.772809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:30.772809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:30.772923 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:30.773265 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:30.773266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:30.773266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:30.773538 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:30.773626 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:30.773979 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 19:03:30.774121 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2451,"thread_start_us":84,"threads_started":1}
I20260925 19:03:30.774369 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:30.761770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:41175)
0925 19:03:30.762365 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:30.762403 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:30.762823 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:30.764810 (+ 1987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:30.764814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:30.765082 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:03:30.765258 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:30.765262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.766467 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.766468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:30.767416 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:30.767419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.767962 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.767963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:30.768103 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:30.768653 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:03:30.768661 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:03:30.770296 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:03:30.772369 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:30.772376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:30.772378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:30.772712 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:30.773037 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:30.773038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:30.773039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:30.773176 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:30.773621 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:03:30.773627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:03:30.773788 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 19:03:30.773903 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 19:03:30.774089 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":85,"threads_started":1}
I20260925 19:03:30.775177 16504 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37199
I20260925 19:03:30.775271 16504 heartbeater.cc:464] Registering TS with master...
I20260925 19:03:30.775431 16504 heartbeater.cc:511] Master 127.0.98.62:37199 requested a full tablet report, sending...
I20260925 19:03:30.776291 16316 ts_manager.cc:194] Registered new tserver with Master: c8bfce2105004a0c8c68de914257f81a (127.0.98.1:42249)
I20260925 19:03:30.777604 16316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41175
I20260925 19:03:30.778574 16504 heartbeater.cc:503] Master 127.0.98.62:37199 was elected leader, sending a full tablet report...
Sep 25 19:03:30 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 19:03:30.785471 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:30.777184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.0.98.1:42249)
0925 19:03:30.777555 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:30.777559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:30.777567 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:03:30.777597 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:30.777599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:30.777613 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:30.777674 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:30.778222 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.779440 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.780171 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.780699 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.780785 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:30.782876 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:30.782882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:30.782883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:30.782889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:30.784320 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:30.784725 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:30.784727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:30.784727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:30.784779 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:30.784954 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:30.784954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:30.784954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:30.785126 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:30.785187 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:30.785345 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 19:03:30.785412 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":343,"thread_start_us":83,"threads_started":1}
I20260925 19:03:30.785526 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:30.776944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42249 (local address 127.0.0.1:40210)
0925 19:03:30.777347 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:30.777382 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:30.777443 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:30.777734 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:30.777738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:30.778069 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:03:30.778141 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:30.778143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.779523 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.779523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:30.780094 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:30.780096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.780779 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.780781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:30.780833 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:30.781363 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:03:30.781367 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:03:30.782794 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:03:30.784394 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:30.784401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:30.784403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:30.784652 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:30.784836 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:30.784837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:30.784837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:30.784900 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:30.785184 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:03:30.785190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:03:30.785238 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 19:03:30.785324 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:03:30.785428 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":89,"threads_started":1}
I20260925 19:03:30.785879 16438 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef5cc287e014469b86d908b5aef1486 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:03:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40210
I20260925 19:03:30.786434 16513 tablet_replica.cc:330] stopping tablet replica
I20260925 19:03:30.786546 16513 raft_consensus.cc:2229] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:03:30.786617 16513 raft_consensus.cc:2258] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:03:30.786871 16513 ts_tablet_manager.cc:1905] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:03:30.788944 16513 ts_tablet_manager.cc:1918] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:03:30.789072 16513 log.cc:1196] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal/wals/2ef5cc287e014469b86d908b5aef1486
I20260925 19:03:30.789299 16513 ts_tablet_manager.cc:1939] T 2ef5cc287e014469b86d908b5aef1486 P c8bfce2105004a0c8c68de914257f81a: Deleting consensus metadata
I20260925 19:03:30.789762 16301 catalog_manager.cc:4841] TS c8bfce2105004a0c8c68de914257f81a (127.0.98.1:42249): tablet 2ef5cc287e014469b86d908b5aef1486 (table test-table [id=279be61c0a9b466999aa2718751da7fe]) successfully deleted
W20260925 19:03:30.853765 16510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:30.853956 16510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:30.853978 16510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:30.855019 16510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:30.855055 16510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:30.855080 16510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:03:30.856376 16510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37303
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:30.856554 16510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:30.856727 16510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:30.861469 16521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:30.861608 16510 server_base.cc:1034] running on GCE node
W20260925 19:03:30.861470 16519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:30.861476 16518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:30.861892 16510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:30.862155 16510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:30.863301 16510 hybrid_clock.cc:648] HybridClock initialized: now 1790363010863279 us; error 33 us; skew 500 ppm
Sep 25 19:03:30 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:30.865729 16510 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:03:30.866209 16510 webserver.cc:460] Webserver started at http://127.0.98.2:37303/ using document root <none> and password file <none>
I20260925 19:03:30.866465 16510 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:30.866537 16510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:30.867808 16510 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:03:30.868366 16528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:30.868525 16510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:03:30.868623 16510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "0a34a864ea3f4030a6b3b30499ec662a"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "c88bb26e8591f0953a8e4f6c583ead16"
server_key_iv: "68b87308a5b86eceb75d7bb858bafb6a"
server_key_version: "encryptionkey@0"
I20260925 19:03:30.868988 16510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:30.892915 16510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:30.894949 16510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:30.895115 16510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:30.895352 16510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:03:30.895648 16510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:03:30.895696 16510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:30.895736 16510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:03:30.895766 16510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:30.901526 16510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40633
I20260925 19:03:30.901592 16641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40633 every 8 connection(s)
I20260925 19:03:30.901901 16510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:03:30.906569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 16510
I20260925 19:03:30.906841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43569
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37199
--webserver_port=46241
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 19:03:30 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:03:30.910723 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:30.902605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56307 (local address 127.0.98.62:37199)
0925 19:03:30.902673 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:30.902675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:30.902860 (+ 185us) server_negotiation.cc:407] Connection header received
0925 19:03:30.903317 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:30.903318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:30.903333 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:30.903372 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:30.904258 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.904706 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.905634 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.905830 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.905880 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:30.907850 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:30.907851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:30.907852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:30.907861 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:30.909304 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:30.909715 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:30.909716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:30.909717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:30.909794 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:30.910013 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:30.910014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:30.910015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:30.910151 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:30.910214 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:30.910499 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 19:03:30.910620 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:03:30.910904 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:30.902603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:56307)
0925 19:03:30.902750 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:30.902781 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:30.903222 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:30.903435 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:30.903438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:30.904017 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:03:30.904194 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:30.904198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.904781 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.904782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:30.905564 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:30.905566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:30.905894 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:30.905895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:30.905981 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:30.906433 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:03:30.906441 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:03:30.907769 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:03:30.909369 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:30.909374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:30.909376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:30.909644 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:30.909871 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:30.909871 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:30.909872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:30.909959 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:30.910235 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:03:30.910241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:03:30.910380 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 19:03:30.910456 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 19:03:30.910627 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260925 19:03:30.911357 16642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37199
I20260925 19:03:30.911451 16642 heartbeater.cc:464] Registering TS with master...
I20260925 19:03:30.911643 16642 heartbeater.cc:511] Master 127.0.98.62:37199 requested a full tablet report, sending...
I20260925 19:03:30.912015 16316 ts_manager.cc:194] Registered new tserver with Master: 0a34a864ea3f4030a6b3b30499ec662a (127.0.98.2:40633)
I20260925 19:03:30.912572 16316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56307
W20260925 19:03:30.988059 16645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:30.988260 16645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:30.988282 16645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:30.989388 16645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:30.989424 16645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:30.989450 16645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:03:30.990797 16645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46241
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:30.990965 16645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:30.991133 16645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:30.995703 16650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:30.995844 16651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:30.995704 16653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:30.995962 16645 server_base.cc:1034] running on GCE node
I20260925 19:03:30.996156 16645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:30.996363 16645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:30.997536 16645 hybrid_clock.cc:648] HybridClock initialized: now 1790363010997505 us; error 33 us; skew 500 ppm
Sep 25 19:03:30 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:31.000036 16645 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:03:31.000514 16645 webserver.cc:460] Webserver started at http://127.0.98.3:46241/ using document root <none> and password file <none>
I20260925 19:03:31.000749 16645 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:31.000829 16645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:31.002170 16645 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:03:31.002810 16660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:31.002981 16645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:31.003078 16645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "d293cc736c514dcf87e50a37157093fb"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "884a430778213a3eca44dbacb4f4c1fb"
server_key_iv: "89d337d13e9ba47d0c6cf7ce34d1058d"
server_key_version: "encryptionkey@0"
I20260925 19:03:31.003408 16645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:31.019640 16645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:31.021477 16645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:31.021626 16645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:31.021834 16645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:03:31.022115 16645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:03:31.022163 16645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:31.022202 16645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:03:31.022222 16645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:31.027793 16645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43569
I20260925 19:03:31.027834 16773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43569 every 8 connection(s)
I20260925 19:03:31.028172 16645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:03:31.032586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 16645
Sep 25 19:03:31 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:03:31.037835 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.028898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57359 (local address 127.0.98.62:37199)
0925 19:03:31.028959 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:31.028961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:31.029152 (+ 191us) server_negotiation.cc:407] Connection header received
0925 19:03:31.029660 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:31.029661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:31.029670 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:31.029731 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:31.030644 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.031084 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.032011 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.032189 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.032244 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.034673 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:31.034674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:31.034674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:31.034682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:31.036262 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:31.036922 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:31.036923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:31.036923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:31.036968 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:31.037244 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:31.037245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:31.037245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:31.037353 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:31.037415 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:31.037667 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 19:03:31.037749 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:03:31.038048 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.028886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:57359)
0925 19:03:31.029028 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:31.029066 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:31.029555 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:31.029799 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:31.029802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:31.030397 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:03:31.030574 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:31.030579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.031173 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.031174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:31.031932 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:31.031934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.032254 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.032255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:31.032358 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.032956 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:03:31.032965 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:03:31.034594 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:03:31.036549 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:31.036555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:31.036558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:31.036850 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:31.037029 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:31.037030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:31.037031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:31.037177 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:31.037424 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:03:31.037429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:03:31.037535 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 19:03:31.037642 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 19:03:31.037785 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260925 19:03:31.038501 16774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37199
I20260925 19:03:31.038615 16774 heartbeater.cc:464] Registering TS with master...
I20260925 19:03:31.038828 16774 heartbeater.cc:511] Master 127.0.98.62:37199 requested a full tablet report, sending...
I20260925 19:03:31.039194 16316 ts_manager.cc:194] Registered new tserver with Master: d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569)
I20260925 19:03:31.039556 16316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57359
I20260925 19:03:31.040179 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.033064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.0.98.62:37199)
0925 19:03:31.033168 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:31.033169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:31.033279 (+ 110us) server_negotiation.cc:407] Connection header received
0925 19:03:31.033342 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:31.033342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:31.033350 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:31.033404 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:31.034128 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.034687 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.035670 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.035970 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.036116 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.037370 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:31.037372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:31.037372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:31.037377 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:31.039017 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:31.039379 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:31.039380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:31.039381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:31.039436 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:31.039653 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:31.039654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:31.039654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:31.039757 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:31.039944 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:31.039978 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 19:03:31.040070 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":61,"threads_started":1}
I20260925 19:03:31.040786 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:03:31.043325 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.041093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.0.98.62:37199)
0925 19:03:31.041129 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:31.041130 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:31.041251 (+ 121us) server_negotiation.cc:407] Connection header received
0925 19:03:31.041284 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:31.041285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:31.041310 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:31.041358 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:03:31.041426 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.041958 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.042691 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.042912 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.042956 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.043083 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:03:31.043118 (+ 35us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 19:03:31.043269 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59962: token expired
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 19:03:31.043380 16777 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59962: token expired
W20260925 19:03:31.043406 16249 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:37199)
I20260925 19:03:31.048781 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.043618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.0.98.62:37199)
0925 19:03:31.043651 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:31.043653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:31.043661 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:03:31.043724 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:31.043725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:31.043733 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:31.043790 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:31.044233 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.044657 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.045364 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.045541 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.045609 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.046693 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:31.046695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:31.046695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:31.046701 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:31.047758 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:31.048072 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:31.048073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:31.048073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:31.048126 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:31.048306 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:31.048307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:31.048308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:31.048401 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:31.048497 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:31.048595 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 19:03:31.048690 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:03:31.049784 16316 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:03:31.051337 16316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:03:31.053432 16438 tablet_service.cc:1467] Processing CreateTablet for tablet 00ad1e33538c4918a20e0114fc70d4e1 (DEFAULT_TABLE table=test-table [id=8abb1b72de974850879b6f94dfffddf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:03:31.053655 16438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ad1e33538c4918a20e0114fc70d4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:03:31.055720 16396 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: Bootstrap starting.
I20260925 19:03:31.056084 16396 tablet_bootstrap.cc:654] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: Neither blocks nor log segments found. Creating new log.
I20260925 19:03:31.056604 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.053024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43569 (local address 127.0.0.1:38256)
0925 19:03:31.053080 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:31.053090 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:31.053135 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:31.053533 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:31.053534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:31.053536 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:03:31.053591 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:31.053592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.054633 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.054634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:31.055596 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:31.055598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.056367 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.056368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:31.056440 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.056460 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 19:03:31.056488 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 19:03:31.056542 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 19:03:31.056660 16396 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: No bootstrap required, opened a new log
I20260925 19:03:31.056708 16396 ts_tablet_manager.cc:1397] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:03:31.056985 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.053196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.0.98.3:43569)
0925 19:03:31.053351 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:31.053355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:31.053364 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:03:31.053400 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:31.053402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:31.053418 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:31.053525 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:03:31.053646 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.054551 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.055679 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.056272 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.056765 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.056833 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:31.056860 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 19:03:31.056901 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260925 19:03:31.056951 16396 raft_consensus.cc:348] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:31.057060 16396 raft_consensus.cc:374] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:03:31.057106 16396 raft_consensus.cc:729] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8bfce2105004a0c8c68de914257f81a, State: Initialized, Role: FOLLOWER
I20260925 19:03:31.057154 16396 consensus_queue.cc:260] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:31.057183 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.053026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.0.1:34484)
0925 19:03:31.053186 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:31.053197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:31.053239 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:31.053727 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:31.053727 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:31.053729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:03:31.053769 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:31.053771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.055034 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.055035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:31.056018 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:31.056021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.056910 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.056911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:31.056984 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.056988 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:03:31.057063 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:03:31.057111 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":27,"thread_start_us":92,"threads_started":1}
I20260925 19:03:31.057204 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.053362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.0.98.2:40633)
0925 19:03:31.053549 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:31.053553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:31.053563 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:03:31.053600 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:31.053602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:31.053615 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:31.053837 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:03:31.053944 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.054951 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.056105 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.056801 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.056894 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.056942 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:31.057086 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 19:03:31.057125 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260925 19:03:31.057248 16396 ts_tablet_manager.cc:1428] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 19:03:31.057900 16708 tablet_service.cc:1467] Processing CreateTablet for tablet 00ad1e33538c4918a20e0114fc70d4e1 (DEFAULT_TABLE table=test-table [id=8abb1b72de974850879b6f94dfffddf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:03:31.058080 16576 tablet_service.cc:1467] Processing CreateTablet for tablet 00ad1e33538c4918a20e0114fc70d4e1 (DEFAULT_TABLE table=test-table [id=8abb1b72de974850879b6f94dfffddf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:03:31.058234 16708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ad1e33538c4918a20e0114fc70d4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:03:31.058403 16576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ad1e33538c4918a20e0114fc70d4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:03:31.060961 16785 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Bootstrap starting.
I20260925 19:03:31.061066 16784 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Bootstrap starting.
I20260925 19:03:31.061488 16785 tablet_bootstrap.cc:654] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Neither blocks nor log segments found. Creating new log.
I20260925 19:03:31.061604 16784 tablet_bootstrap.cc:654] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Neither blocks nor log segments found. Creating new log.
I20260925 19:03:31.061971 16785 log.cc:824] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:31.062063 16784 log.cc:824] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:31.062747 16785 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: No bootstrap required, opened a new log
I20260925 19:03:31.062815 16785 ts_tablet_manager.cc:1397] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:03:31.062899 16784 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: No bootstrap required, opened a new log
I20260925 19:03:31.062997 16784 ts_tablet_manager.cc:1397] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:03:31.064177 16785 raft_consensus.cc:348] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:31.064342 16785 raft_consensus.cc:374] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:03:31.064266 16784 raft_consensus.cc:348] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:31.064388 16785 raft_consensus.cc:729] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a34a864ea3f4030a6b3b30499ec662a, State: Initialized, Role: FOLLOWER
I20260925 19:03:31.064416 16784 raft_consensus.cc:374] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:03:31.064457 16784 raft_consensus.cc:729] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d293cc736c514dcf87e50a37157093fb, State: Initialized, Role: FOLLOWER
I20260925 19:03:31.064495 16785 consensus_queue.cc:260] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:31.064606 16784 consensus_queue.cc:260] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:31.064846 16642 heartbeater.cc:503] Master 127.0.98.62:37199 was elected leader, sending a full tablet report...
I20260925 19:03:31.064893 16784 ts_tablet_manager.cc:1428] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 19:03:31.064918 16774 heartbeater.cc:503] Master 127.0.98.62:37199 was elected leader, sending a full tablet report...
I20260925 19:03:31.064831 16785 ts_tablet_manager.cc:1428] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260925 19:03:31.152551 16643 tablet.cc:2369] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:03:31.217417 16789 raft_consensus.cc:482] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:03:31.217548 16789 raft_consensus.cc:504] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:31.217912 16789 leader_election.cc:288] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a34a864ea3f4030a6b3b30499ec662a (127.0.98.2:40633), c8bfce2105004a0c8c68de914257f81a (127.0.98.1:42249)
I20260925 19:03:31.221732 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.218140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.98.3:47923)
0925 19:03:31.218192 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:31.218204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:31.218268 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:31.218456 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:31.218457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:31.218459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:03:31.218552 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:31.218554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.219497 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.219497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:31.220721 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:31.220723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.221479 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.221480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:31.221566 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.221568 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:03:31.221619 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 19:03:31.221647 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":8}
I20260925 19:03:31.221738 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.218290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42249 (local address 127.0.98.3:36153)
0925 19:03:31.218446 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:31.218456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:31.218508 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:31.218698 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:31.218699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:31.218700 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:03:31.218747 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:31.218748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.219592 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.219592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:31.220721 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:31.220723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.221479 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.221480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:31.221558 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.221562 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:03:31.221608 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 19:03:31.221646 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":18,"thread_start_us":108,"threads_started":1}
I20260925 19:03:31.221781 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.218377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36153 (local address 127.0.98.1:42249)
0925 19:03:31.218421 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:31.218424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:31.218512 (+ 88us) server_negotiation.cc:407] Connection header received
0925 19:03:31.218565 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:31.218566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:31.218602 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:31.218689 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:03:31.218803 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.219522 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.220817 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.221406 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.221477 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.221638 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:31.221667 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 19:03:31.221711 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:03:31.221779 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.218234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47923 (local address 127.0.98.2:40633)
0925 19:03:31.218311 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:31.218313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:31.218324 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:03:31.218356 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:31.218357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:31.218368 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:31.218441 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:03:31.218627 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.219417 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.220817 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.221383 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.221468 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.221479 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:31.221672 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 19:03:31.221715 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 19:03:31.222065 16458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad1e33538c4918a20e0114fc70d4e1" candidate_uuid: "d293cc736c514dcf87e50a37157093fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bfce2105004a0c8c68de914257f81a" is_pre_election: true
I20260925 19:03:31.222098 16596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad1e33538c4918a20e0114fc70d4e1" candidate_uuid: "d293cc736c514dcf87e50a37157093fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a34a864ea3f4030a6b3b30499ec662a" is_pre_election: true
I20260925 19:03:31.222237 16458 raft_consensus.cc:2454] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d293cc736c514dcf87e50a37157093fb in term 0.
I20260925 19:03:31.222242 16596 raft_consensus.cc:2454] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d293cc736c514dcf87e50a37157093fb in term 0.
I20260925 19:03:31.222527 16662 leader_election.cc:302] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8bfce2105004a0c8c68de914257f81a, d293cc736c514dcf87e50a37157093fb; no voters:
I20260925 19:03:31.222681 16789 raft_consensus.cc:2781] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:03:31.222746 16789 raft_consensus.cc:482] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:03:31.222764 16789 raft_consensus.cc:3037] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:03:31.223523 16789 raft_consensus.cc:504] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:31.223658 16789 leader_election.cc:288] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [CANDIDATE]: Term 1 election: Requested vote from peers 0a34a864ea3f4030a6b3b30499ec662a (127.0.98.2:40633), c8bfce2105004a0c8c68de914257f81a (127.0.98.1:42249)
I20260925 19:03:31.223850 16596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad1e33538c4918a20e0114fc70d4e1" candidate_uuid: "d293cc736c514dcf87e50a37157093fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a34a864ea3f4030a6b3b30499ec662a"
I20260925 19:03:31.223858 16458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad1e33538c4918a20e0114fc70d4e1" candidate_uuid: "d293cc736c514dcf87e50a37157093fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bfce2105004a0c8c68de914257f81a"
I20260925 19:03:31.223912 16596 raft_consensus.cc:3037] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:03:31.223953 16458 raft_consensus.cc:3037] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:03:31.224623 16596 raft_consensus.cc:2454] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d293cc736c514dcf87e50a37157093fb in term 1.
I20260925 19:03:31.224678 16458 raft_consensus.cc:2454] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d293cc736c514dcf87e50a37157093fb in term 1.
I20260925 19:03:31.224835 16662 leader_election.cc:302] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8bfce2105004a0c8c68de914257f81a, d293cc736c514dcf87e50a37157093fb; no voters:
I20260925 19:03:31.224915 16789 raft_consensus.cc:2781] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:03:31.225050 16789 raft_consensus.cc:686] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 1 LEADER]: Becoming Leader. State: Replica: d293cc736c514dcf87e50a37157093fb, State: Running, Role: LEADER
I20260925 19:03:31.225136 16789 consensus_queue.cc:237] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:31.225898 16315 catalog_manager.cc:5495] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb reported cstate change: term changed from 0 to 1, leader changed from <none> to d293cc736c514dcf87e50a37157093fb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d293cc736c514dcf87e50a37157093fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } health_report { overall_health: UNKNOWN } } }
I20260925 19:03:31.238027 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.235799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.0.98.3:43569)
0925 19:03:31.235867 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:31.235869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:31.235879 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:03:31.235941 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:31.235941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:31.235961 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:31.236044 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:03:31.236179 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.236654 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.237328 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:31.237525 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:31.237591 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:31.237718 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:03:31.237810 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:03:31.237893 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:31.237915 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 19:03:31.237932 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 19:03:31.239684 16458 raft_consensus.cc:1261] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Refusing update from remote peer d293cc736c514dcf87e50a37157093fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:03:31.239704 16596 raft_consensus.cc:1261] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Refusing update from remote peer d293cc736c514dcf87e50a37157093fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:03:31.239996 16791 consensus_queue.cc:1035] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:03:31.240116 16789 consensus_queue.cc:1035] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:03:31.242162 16794 mvcc.cc:205] Tried to move back new op lower bound from 7333326894035357696 to 7333326893978750976. Current Snapshot: MvccSnapshot[applied={T|T < 7333326894035357696}]
I20260925 19:03:31.242501 16795 mvcc.cc:205] Tried to move back new op lower bound from 7333326894035357696 to 7333326893978750976. Current Snapshot: MvccSnapshot[applied={T|T < 7333326894035357696}]
I20260925 19:03:31.242628 16796 mvcc.cc:205] Tried to move back new op lower bound from 7333326894035357696 to 7333326893978750976. Current Snapshot: MvccSnapshot[applied={T|T < 7333326894035357696}]
I20260925 19:03:31.244375 16316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:03:31.244462 16316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:03:31.245082 16316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Sending DeleteTablet for 3 replicas of tablet 00ad1e33538c4918a20e0114fc70d4e1
I20260925 19:03:31.245355 16708 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ad1e33538c4918a20e0114fc70d4e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:03:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38256
I20260925 19:03:31.245385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 16354
I20260925 19:03:31.245391 16438 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ad1e33538c4918a20e0114fc70d4e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:03:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40210
I20260925 19:03:31.245471 16513 tablet_replica.cc:330] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: stopping tablet replica
I20260925 19:03:31.245529 16513 raft_consensus.cc:2229] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:03:31.245569 16513 raft_consensus.cc:2258] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:03:31.247139 16804 tablet_replica.cc:330] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: stopping tablet replica
I20260925 19:03:31.247155 16576 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ad1e33538c4918a20e0114fc70d4e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:03:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34484
I20260925 19:03:31.247251 16804 raft_consensus.cc:2229] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:03:31.247356 16804 raft_consensus.cc:2258] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:03:31.247756 16804 ts_tablet_manager.cc:1905] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:03:31.249713 16804 ts_tablet_manager.cc:1918] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:03:31.249787 16804 log.cc:1196] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/00ad1e33538c4918a20e0114fc70d4e1
I20260925 19:03:31.250085 16804 ts_tablet_manager.cc:1939] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Deleting consensus metadata
I20260925 19:03:31.250388 16805 tablet_replica.cc:330] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: stopping tablet replica
I20260925 19:03:31.250486 16300 catalog_manager.cc:4841] TS d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569): tablet 00ad1e33538c4918a20e0114fc70d4e1 (table test-table [id=8abb1b72de974850879b6f94dfffddf4]) successfully deleted
I20260925 19:03:31.250602 16805 raft_consensus.cc:2229] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:03:31.250730 16805 raft_consensus.cc:2258] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:03:31.251121 16805 ts_tablet_manager.cc:1905] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 19:03:31.251435 16301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42249 (error 108)
I20260925 19:03:31.251875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 16510
I20260925 19:03:31.252017 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.251881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42249 (local address 127.0.0.1:40220)
0925 19:03:31.251932 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:31.251980 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 19:03:31.253053 16301 catalog_manager.cc:4584] TS c8bfce2105004a0c8c68de914257f81a (127.0.98.1:42249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00ad1e33538c4918a20e0114fc70d4e1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42249: connect: Connection refused (error 111)
I20260925 19:03:31.257091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 16645
I20260925 19:03:31.257131 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:31.257036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.0.1:34486)
0925 19:03:31.257076 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:31.257100 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 19:03:31.261607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 16283
I20260925 19:03:51.267127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37199
--webserver_interface=127.0.98.62
--webserver_port=39649
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:03:51.345983 16807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:51.346220 16807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:51.346256 16807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:51.347342 16807 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 19:03:51.347409 16807 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 19:03:51.347429 16807 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 19:03:51.347445 16807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:03:51.347465 16807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:51.347496 16807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:51.347519 16807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:03:51.347548 16807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:03:51.348989 16807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:51.349259 16807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:51.349480 16807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:03:51.354377 16807 server_base.cc:1034] running on GCE node
W20260925 19:03:51.354481 16812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:51.354434 16813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:51.354439 16815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:51.354959 16807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:51.355240 16807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:51.356390 16807 hybrid_clock.cc:648] HybridClock initialized: now 1790363031356366 us; error 41 us; skew 500 ppm
Sep 25 19:03:51 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:51.364620 16807 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:03:51.365025 16807 webserver.cc:460] Webserver started at http://127.0.98.62:39649/ using document root <none> and password file <none>
I20260925 19:03:51.365214 16807 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:51.365255 16807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:51.366539 16807 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:03:51.367153 16822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:51.367337 16807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:03:51.367417 16807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "245c2fc6202348f3b5ebce7a7debb3c5"
format_stamp: "Formatted at 2026-09-25 19:03:09 on dist-test-slave-49bz"
server_key: "470cab1b7dbfcfdcee233cff9316a4f2"
server_key_iv: "e31c5c58e2d4c17545f87d16bb900e14"
server_key_version: "encryptionkey@0"
I20260925 19:03:51.367740 16807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:51.383759 16807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:51.386027 16807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:51.386134 16807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:51.389905 16807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37199
I20260925 19:03:51.389978 16874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37199 every 8 connection(s)
I20260925 19:03:51.390332 16807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:03:51.391846 16875 sys_catalog.cc:263] Verifying existing consensus state
I20260925 19:03:51.392005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 16807
I20260925 19:03:51.392297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42249
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37199
--webserver_port=34435
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:03:51.392848 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Bootstrap starting.
I20260925 19:03:51.401573 16875 log.cc:824] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:51.403818 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:03:51.404117 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Bootstrap complete.
I20260925 19:03:51.406437 16875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:51.406672 16875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245c2fc6202348f3b5ebce7a7debb3c5, State: Initialized, Role: FOLLOWER
I20260925 19:03:51.406864 16875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:51.406977 16875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:03:51.407027 16875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:03:51.407078 16875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 19:03:51.408023 16875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:51.408147 16875 leader_election.cc:302] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 245c2fc6202348f3b5ebce7a7debb3c5; no voters:
I20260925 19:03:51.408340 16875 leader_election.cc:288] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 19:03:51.408375 16879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 19:03:51.408497 16879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [term 3 LEADER]: Becoming Leader. State: Replica: 245c2fc6202348f3b5ebce7a7debb3c5, State: Running, Role: LEADER
I20260925 19:03:51.408604 16875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:03:51.408644 16879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } }
I20260925 19:03:51.408913 16879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 245c2fc6202348f3b5ebce7a7debb3c5. Latest consensus state: current_term: 3 leader_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } } }
I20260925 19:03:51.408999 16879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:03:51.408918 16880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "245c2fc6202348f3b5ebce7a7debb3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37199 } } }
I20260925 19:03:51.409145 16880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5 [sys.catalog]: This master's current role is: LEADER
I20260925 19:03:51.409204 16885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:03:51.410089 16885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:03:51.410195 16885 catalog_manager.cc:1259] Loaded cluster ID: 1edcd605ca0741168f6fdb0c303ad99d
I20260925 19:03:51.410231 16885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:03:51.411517 16885 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:03:51.411687 16885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Loaded TSK: 0
I20260925 19:03:51.411734 16885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 245c2fc6202348f3b5ebce7a7debb3c5: Loaded TSK: 1
F20260925 19:03:51.411952 16885 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff4636389ad google::LogMessage::Fail() at ??:0
@ 0x7ff46363b88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff4636384de google::LogMessage::Flush() at ??:0
@ 0x7ff463639a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff465da5104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff465da2d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff46b5921cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff46b59248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff46b592594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff46b558c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff46b562ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff4655060d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff4654fb954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff46635f6db start_thread at ??:0
@ 0x7ff46213661f clone at ??:0
W20260925 19:03:51.489764 16877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:51.489967 16877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:51.489991 16877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:51.491235 16877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:51.491277 16877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:51.491303 16877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:03:51.492764 16877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34435
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:51.492992 16877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:51.493223 16877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:51.499199 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:51.499215 16916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:51.499387 16877 server_base.cc:1034] running on GCE node
W20260925 19:03:51.499257 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:51.499612 16877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:51.499857 16877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:51.501014 16877 hybrid_clock.cc:648] HybridClock initialized: now 1790363031500999 us; error 33 us; skew 500 ppm
Sep 25 19:03:51 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:51.503466 16877 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:03:51.503866 16877 webserver.cc:460] Webserver started at http://127.0.98.1:34435/ using document root <none> and password file <none>
I20260925 19:03:51.504087 16877 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:51.504153 16877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:51.505493 16877 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:03:51.506217 16926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:51.506388 16877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:03:51.506488 16877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "c8bfce2105004a0c8c68de914257f81a"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "76dcebb7723d42d2ccd560298a48023e"
server_key_iv: "a0b273a860d249f51b4362c5b62f25bb"
server_key_version: "encryptionkey@0"
I20260925 19:03:51.506832 16877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:51.521086 16877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:51.523411 16877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:51.523566 16877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:51.523799 16877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:03:51.524258 16933 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:03:51.526310 16877 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:03:51.526376 16877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:03:51.526437 16877 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:03:51.527063 16877 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:03:51.527117 16877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:03:51.527189 16933 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: Bootstrap starting.
I20260925 19:03:51.534359 16877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42249
I20260925 19:03:51.534513 17040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42249 every 8 connection(s)
I20260925 19:03:51.534776 16877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:03:51.537674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 16877
I20260925 19:03:51.537968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40633
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37199
--webserver_port=37303
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:03:51.538530 16933 log.cc:824] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:51.541244 16933 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:03:51.541575 16933 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: Bootstrap complete.
I20260925 19:03:51.542814 16933 ts_tablet_manager.cc:1397] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260925 19:03:51.544296 16933 raft_consensus.cc:348] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:51.544646 16933 raft_consensus.cc:729] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8bfce2105004a0c8c68de914257f81a, State: Initialized, Role: FOLLOWER
I20260925 19:03:51.544813 16933 consensus_queue.cc:260] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:51.545076 16933 ts_tablet_manager.cc:1428] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260925 19:03:51.609275 17043 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:03:51.535441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:55433)
0925 19:03:51.535622 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:51.535666 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:51.536154 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:51.609072 (+ 72918us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":113,"threads_started":1}
W20260925 19:03:51.609553 17041 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37199 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37199: Client connection negotiation failed: client connection to 127.0.98.62:37199: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 19:03:51.609901 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:51.609790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:36683)
0925 19:03:51.609842 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:51.609882 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 19:03:51.610285 17043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:51.610142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:37231)
0925 19:03:51.610185 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:51.610210 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 19:03:51.610376 17041 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 19:03:51.619614 17045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:51.619817 17045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:51.619863 17045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:51.621004 17045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:51.621066 17045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:51.621110 17045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:03:51.622776 17045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37303
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:51.623031 17045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:51.623294 17045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:51.628355 17052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:51.628422 17045 server_base.cc:1034] running on GCE node
W20260925 19:03:51.628363 17051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:51.628347 17054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:51.628749 17045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:51.628990 17045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:51.630132 17045 hybrid_clock.cc:648] HybridClock initialized: now 1790363031630115 us; error 34 us; skew 500 ppm
Sep 25 19:03:51 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:51.632999 17045 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:03:51.633394 17045 webserver.cc:460] Webserver started at http://127.0.98.2:37303/ using document root <none> and password file <none>
I20260925 19:03:51.633618 17045 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:51.633663 17045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:51.634941 17045 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260925 19:03:51.635612 17061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:51.635771 17045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:03:51.635849 17045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "0a34a864ea3f4030a6b3b30499ec662a"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "c88bb26e8591f0953a8e4f6c583ead16"
server_key_iv: "68b87308a5b86eceb75d7bb858bafb6a"
server_key_version: "encryptionkey@0"
I20260925 19:03:51.636134 17045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:51.646876 17045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:51.648572 17045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:51.648674 17045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:51.648846 17045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:03:51.649282 17068 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 19:03:51.650710 17045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 19:03:51.650781 17045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 19:03:51.650803 17045 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 19:03:51.651387 17045 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 19:03:51.651418 17045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 19:03:51.651525 17068 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Bootstrap starting.
I20260925 19:03:51.653275 17068 log.cc:824] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:51.656229 17068 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:03:51.656505 17068 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Bootstrap complete.
I20260925 19:03:51.656648 17068 ts_tablet_manager.cc:1397] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 19:03:51.657711 17045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40633
I20260925 19:03:51.657775 17176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40633 every 8 connection(s)
I20260925 19:03:51.658042 17045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:03:51.658463 17068 raft_consensus.cc:348] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:51.658736 17068 raft_consensus.cc:729] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a34a864ea3f4030a6b3b30499ec662a, State: Initialized, Role: FOLLOWER
I20260925 19:03:51.658888 17068 consensus_queue.cc:260] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:51.659138 17068 ts_tablet_manager.cc:1428] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260925 19:03:51.659148 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:51.658721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:43169)
0925 19:03:51.658887 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:51.658991 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
W20260925 19:03:51.659353 17177 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37199 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37199: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
I20260925 19:03:51.659673 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:51.659578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:51825)
0925 19:03:51.659621 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:51.659644 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 19:03:51.659905 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:51.659861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:56469)
0925 19:03:51.659880 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:51.659895 (+ 15us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260925 19:03:51.660002 17177 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:03:51.663058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 17045
I20260925 19:03:51.663327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43569
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37199
--webserver_port=46241
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:03:51.744622 17181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:03:51.744817 17181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:03:51.744838 17181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:03:51.745870 17181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:03:51.745910 17181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:03:51.745935 17181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:03:51.747380 17181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46241
--tserver_master_addrs=127.0.98.62:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:03:51.747560 17181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:03:51.747715 17181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:03:51.752177 17189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:51.752198 17186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:03:51.752180 17187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:03:51.752452 17181 server_base.cc:1034] running on GCE node
I20260925 19:03:51.752657 17181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:03:51.752835 17181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:03:51.753973 17181 hybrid_clock.cc:648] HybridClock initialized: now 1790363031753962 us; error 30 us; skew 500 ppm
Sep 25 19:03:51 dist-test-slave-49bz krb5kdc[15740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:03:51.756307 17181 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:03:51.756763 17181 webserver.cc:460] Webserver started at http://127.0.98.3:46241/ using document root <none> and password file <none>
I20260925 19:03:51.756934 17181 fs_manager.cc:359] Metadata directory not provided
I20260925 19:03:51.756976 17181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:03:51.758167 17181 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:03:51.758756 17196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:51.758929 17181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:51.758996 17181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "d293cc736c514dcf87e50a37157093fb"
format_stamp: "Formatted at 2026-09-25 19:03:10 on dist-test-slave-49bz"
server_key: "884a430778213a3eca44dbacb4f4c1fb"
server_key_iv: "89d337d13e9ba47d0c6cf7ce34d1058d"
server_key_version: "encryptionkey@0"
I20260925 19:03:51.759356 17181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:03:51.772516 17181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:03:51.774379 17181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:03:51.774488 17181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:03:51.774732 17181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:03:51.775019 17181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:03:51.775068 17181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:51.775108 17181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:03:51.775134 17181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:03:51.780809 17181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43569
I20260925 19:03:51.780894 17309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43569 every 8 connection(s)
I20260925 19:03:51.781209 17181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:03:51.782230 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:51.781839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:56317)
0925 19:03:51.782035 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:51.782100 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
W20260925 19:03:51.782657 17310 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37199 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37199: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
I20260925 19:03:51.782889 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:51.782826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:57005)
0925 19:03:51.782852 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:51.782875 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 19:03:51.783110 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:51.783063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:52711)
0925 19:03:51.783082 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:51.783099 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 19:03:51.783206 17310 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 19:03:51.787961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 17181
I20260925 19:03:52.610965 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:52.610740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:60763)
0925 19:03:52.610861 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:52.610896 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260925 19:03:52.660459 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:52.660263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:59803)
0925 19:03:52.660379 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:52.660412 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260925 19:03:52.783677 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:52.783488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:53657)
0925 19:03:52.783596 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:52.783626 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260925 19:03:52.993127 17317 raft_consensus.cc:482] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:03:52.993242 17317 raft_consensus.cc:504] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:52.993539 17317 leader_election.cc:288] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569), c8bfce2105004a0c8c68de914257f81a (127.0.98.1:42249)
Sep 25 19:03:52 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
Sep 25 19:03:52 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 19:03:53.003386 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:52.993725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42249 (local address 127.0.98.2:39797)
0925 19:03:52.993850 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:52.993859 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:52.993911 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:52.994242 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:52.994246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:52.994659 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:03:52.995107 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:52.995110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:52.996204 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:52.996204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:52.996924 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:52.996925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:52.997470 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:52.997471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:52.997586 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:52.998120 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:03:52.998127 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:03:52.999568 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:03:53.001713 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:53.001719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:53.001721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:53.002042 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:53.002353 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:53.002354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:53.002355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:53.002465 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:53.002893 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:03:53.002894 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:03:53.003009 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 19:03:53.003109 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 19:03:53.003288 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":30,"thread_start_us":65,"threads_started":1}
I20260925 19:03:53.003417 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:52.993799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55113 (local address 127.0.98.3:43569)
0925 19:03:52.993937 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:52.993949 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:52.993979 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:03:52.994603 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:52.994606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:52.994652 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:52.994775 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:52.995203 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:52.996054 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:52.997043 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:52.997379 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:52.997505 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:53.000074 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:53.000077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:53.000077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:53.000086 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:53.001792 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:53.002179 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:53.002182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:53.002182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:53.002301 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:53.002541 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:53.002542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:53.002543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:53.002797 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:53.002877 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:53.003172 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 19:03:53.003288 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":46,"threads_started":1}
I20260925 19:03:53.003437 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:52.993838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39797 (local address 127.0.98.1:42249)
0925 19:03:52.993994 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:52.993998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:52.994009 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:03:52.994089 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:52.994090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:52.994107 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:52.994170 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:52.995246 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:52.996086 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:52.997011 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:52.997380 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:52.997504 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:52.999677 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:52.999680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:52.999681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:52.999693 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:53.001634 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:53.002131 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:53.002135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:53.002135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:53.002294 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:53.002537 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:53.002538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:53.002538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:53.002804 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:53.002898 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:53.003177 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 19:03:53.003319 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
I20260925 19:03:53.003540 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:52.993667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43569 (local address 127.0.98.2:55113)
0925 19:03:52.993702 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:52.993727 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:52.993901 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:52.994783 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:52.994784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:52.994936 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:03:52.995081 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:52.995086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:52.996151 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:52.996152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:52.996934 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:52.996937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:52.997467 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:52.997468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:52.997585 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:52.998187 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:03:52.998191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:03:52.999958 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:03:53.001870 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:53.001871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:53.001871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:53.002104 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:53.002363 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:53.002364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:53.002364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:53.002468 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:53.002868 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:03:53.002874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:03:53.003009 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 19:03:53.003108 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 19:03:53.003364 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":168}
I20260925 19:03:53.004537 16995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad1e33538c4918a20e0114fc70d4e1" candidate_uuid: "0a34a864ea3f4030a6b3b30499ec662a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c8bfce2105004a0c8c68de914257f81a" is_pre_election: true
I20260925 19:03:53.004703 16995 raft_consensus.cc:2454] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a34a864ea3f4030a6b3b30499ec662a in term 1.
W20260925 19:03:53.004917 17062 leader_election.cc:341] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569): Not found: Tablet not found: 00ad1e33538c4918a20e0114fc70d4e1
I20260925 19:03:53.005113 17063 leader_election.cc:302] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a34a864ea3f4030a6b3b30499ec662a, c8bfce2105004a0c8c68de914257f81a; no voters: d293cc736c514dcf87e50a37157093fb
I20260925 19:03:53.005216 17317 raft_consensus.cc:2781] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 19:03:53.005270 17317 raft_consensus.cc:482] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:03:53.005296 17317 raft_consensus.cc:3037] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 1 FOLLOWER]: Advancing to term 2
I20260925 19:03:53.006497 17317 raft_consensus.cc:504] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:53.006654 17317 leader_election.cc:288] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [CANDIDATE]: Term 2 election: Requested vote from peers d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569), c8bfce2105004a0c8c68de914257f81a (127.0.98.1:42249)
I20260925 19:03:53.006821 16995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad1e33538c4918a20e0114fc70d4e1" candidate_uuid: "0a34a864ea3f4030a6b3b30499ec662a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c8bfce2105004a0c8c68de914257f81a"
I20260925 19:03:53.006902 16995 raft_consensus.cc:3037] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 1 FOLLOWER]: Advancing to term 2
W20260925 19:03:53.006959 17062 leader_election.cc:341] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569): Not found: Tablet not found: 00ad1e33538c4918a20e0114fc70d4e1
I20260925 19:03:53.008396 16995 raft_consensus.cc:2454] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a34a864ea3f4030a6b3b30499ec662a in term 2.
I20260925 19:03:53.008589 17063 leader_election.cc:302] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a34a864ea3f4030a6b3b30499ec662a, c8bfce2105004a0c8c68de914257f81a; no voters: d293cc736c514dcf87e50a37157093fb
I20260925 19:03:53.008709 17317 raft_consensus.cc:2781] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 2 FOLLOWER]: Leader election won for term 2
I20260925 19:03:53.008847 17317 raft_consensus.cc:686] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [term 2 LEADER]: Becoming Leader. State: Replica: 0a34a864ea3f4030a6b3b30499ec662a, State: Running, Role: LEADER
I20260925 19:03:53.008946 17317 consensus_queue.cc:237] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:53.009320 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:53.009197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:39533)
0925 19:03:53.009238 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:53.009264 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 19:03:53.483038 16995 raft_consensus.cc:1261] T 00ad1e33538c4918a20e0114fc70d4e1 P c8bfce2105004a0c8c68de914257f81a [term 2 FOLLOWER]: Refusing update from remote peer 0a34a864ea3f4030a6b3b30499ec662a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 19:03:53.483394 17317 consensus_queue.cc:1035] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 19:03:53.484336 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:53.484165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:56429)
0925 19:03:53.484261 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:53.484302 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
I20260925 19:03:53.485122 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:53.485024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:41901)
0925 19:03:53.485066 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:53.485091 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 19:03:53.485216 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:53.485111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:60117)
0925 19:03:53.485148 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:53.485186 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 19:03:53.485311 17062 consensus_peers.cc:487] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a -> Peer d293cc736c514dcf87e50a37157093fb (127.0.98.3:43569): Couldn't send request to peer d293cc736c514dcf87e50a37157093fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00ad1e33538c4918a20e0114fc70d4e1. This is attempt 1: this message will repeat every 5th retry.
I20260925 19:03:53.784240 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:53.784061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:51885)
0925 19:03:53.784173 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:53.784205 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260925 19:03:54.095719 17330 ts_tablet_manager.cc:927] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Initiating tablet copy from peer 0a34a864ea3f4030a6b3b30499ec662a (127.0.98.2:40633)
I20260925 19:03:54.095963 17330 tablet_copy_client.cc:316] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:40633
Sep 25 19:03:54 dist-test-slave-49bz krb5kdc[15740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260925 19:03:54.103210 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:54.096343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38981 (local address 127.0.98.2:40633)
0925 19:03:54.096453 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 19:03:54.096457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:03:54.096466 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:03:54.096489 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:03:54.096491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:03:54.096503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:03:54.096589 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:03:54.097038 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:54.097720 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:54.098492 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:03:54.098785 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:03:54.098839 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:54.100744 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:03:54.100749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:03:54.100750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:03:54.100755 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:03:54.102142 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:54.102536 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:54.102538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:54.102538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:54.102589 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:03:54.102750 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:03:54.102750 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:03:54.102750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:03:54.102877 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:03:54.102934 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:03:54.103090 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 19:03:54.103147 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":47,"threads_started":1}
I20260925 19:03:54.103216 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:54.096255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40633 (local address 127.0.98.3:38981)
0925 19:03:54.096317 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:54.096335 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 19:03:54.096397 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:03:54.096602 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:03:54.096606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:03:54.096870 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:03:54.096942 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:54.096943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:54.097792 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:54.097793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:54.098427 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:03:54.098429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:03:54.098849 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:03:54.098850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:03:54.098901 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:03:54.099432 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:03:54.099436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:03:54.100676 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:03:54.102213 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:54.102218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:54.102220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:54.102452 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:54.102646 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:03:54.102647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:03:54.102647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:03:54.102700 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:03:54.102932 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:03:54.102937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:03:54.102981 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 19:03:54.103052 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 19:03:54.103138 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 19:03:54.103713 17151 tablet_copy_service.cc:140] P 0a34a864ea3f4030a6b3b30499ec662a: Received BeginTabletCopySession request for tablet 00ad1e33538c4918a20e0114fc70d4e1 from peer d293cc736c514dcf87e50a37157093fb ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38981)
I20260925 19:03:54.103775 17151 tablet_copy_service.cc:161] P 0a34a864ea3f4030a6b3b30499ec662a: Beginning new tablet copy session on tablet 00ad1e33538c4918a20e0114fc70d4e1 from peer d293cc736c514dcf87e50a37157093fb at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38981: session id = d293cc736c514dcf87e50a37157093fb-00ad1e33538c4918a20e0114fc70d4e1
I20260925 19:03:54.104288 17151 tablet_copy_source_session.cc:205] T 00ad1e33538c4918a20e0114fc70d4e1 P 0a34a864ea3f4030a6b3b30499ec662a: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 19:03:54.105298 17330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ad1e33538c4918a20e0114fc70d4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:03:54.107841 17330 tablet_copy_client.cc:791] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: tablet copy: Starting download of 0 data blocks...
I20260925 19:03:54.107990 17330 tablet_copy_client.cc:655] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: tablet copy: Starting download of 1 WAL segments...
I20260925 19:03:54.108781 17330 tablet_copy_client.cc:531] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 19:03:54.109577 17330 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Bootstrap starting.
I20260925 19:03:54.111074 17330 log.cc:824] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Log is configured to *not* fsync() on all Append() calls
I20260925 19:03:54.113297 17330 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 19:03:54.113512 17330 tablet_bootstrap.cc:492] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Bootstrap complete.
I20260925 19:03:54.113646 17330 ts_tablet_manager.cc:1397] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 19:03:54.114624 17330 raft_consensus.cc:348] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:54.114791 17330 raft_consensus.cc:729] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d293cc736c514dcf87e50a37157093fb, State: Initialized, Role: FOLLOWER
I20260925 19:03:54.114921 17330 consensus_queue.cc:260] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d293cc736c514dcf87e50a37157093fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43569 } } peers { permanent_uuid: "0a34a864ea3f4030a6b3b30499ec662a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40633 } } peers { permanent_uuid: "c8bfce2105004a0c8c68de914257f81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42249 } }
I20260925 19:03:54.115183 17330 ts_tablet_manager.cc:1428] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20260925 19:03:54.115484 17151 tablet_copy_service.cc:342] P 0a34a864ea3f4030a6b3b30499ec662a: Request end of tablet copy session d293cc736c514dcf87e50a37157093fb-00ad1e33538c4918a20e0114fc70d4e1 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38981
I20260925 19:03:54.115543 17151 tablet_copy_service.cc:433] P 0a34a864ea3f4030a6b3b30499ec662a: ending tablet copy session d293cc736c514dcf87e50a37157093fb-00ad1e33538c4918a20e0114fc70d4e1 on tablet 00ad1e33538c4918a20e0114fc70d4e1 with peer d293cc736c514dcf87e50a37157093fb
I20260925 19:03:54.115545 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:54.115453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:43395)
0925 19:03:54.115485 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:54.115517 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 19:03:54.115845 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:54.115731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:44903)
0925 19:03:54.115787 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:54.115817 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 19:03:54.485801 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:54.485592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:51621)
0925 19:03:54.485721 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:54.485756 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260925 19:03:54.485822 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:54.485567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:39083)
0925 19:03:54.485741 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:54.485767 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260925 19:03:54.606837 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:54.606708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:35303)
0925 19:03:54.606762 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:54.606798 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 19:03:54.606909 17264 raft_consensus.cc:1203] T 00ad1e33538c4918a20e0114fc70d4e1 P d293cc736c514dcf87e50a37157093fb [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 19:03:54.607393 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:54.607239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:56667)
0925 19:03:54.607309 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:54.607364 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 19:03:55.486438 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:55.486204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:55729)
0925 19:03:55.486364 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:55.486394 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260925 19:03:55.607605 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:55.607408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:60665)
0925 19:03:55.607525 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:55.607560 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260925 19:03:55.607931 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:55.607754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:46243)
0925 19:03:55.607859 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:55.607894 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260925 19:03:56.487103 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:56.486878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:40363)
0925 19:03:56.487017 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:56.487059 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260925 19:03:56.608297 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:56.608042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:43935)
0925 19:03:56.608183 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:56.608238 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260925 19:03:56.608651 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:56.608306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:34625)
0925 19:03:56.608563 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:56.608613 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":51,"threads_started":1}
I20260925 19:03:57.487773 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:57.487568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:51473)
0925 19:03:57.487692 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:57.487728 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260925 19:03:57.608950 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:57.608734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:42427)
0925 19:03:57.608873 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:57.608907 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":54,"threads_started":1}
I20260925 19:03:57.609369 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:57.609260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:52367)
0925 19:03:57.609321 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:57.609347 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 19:03:57.609397 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:57.609184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:57991)
0925 19:03:57.609332 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:57.609364 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260925 19:03:58.488394 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:58.488180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:47797)
0925 19:03:58.488310 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:58.488349 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260925 19:03:58.610040 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:58.609796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:52309)
0925 19:03:58.609952 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:58.609988 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260925 19:03:58.610054 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:58.609781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:34993)
0925 19:03:58.609904 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:58.609964 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260925 19:03:59.489094 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:59.488845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:53019)
0925 19:03:59.489017 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:59.489049 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260925 19:03:59.610716 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:59.610462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:45513)
0925 19:03:59.610640 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:59.610676 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260925 19:03:59.610723 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:03:59.610468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:49767)
0925 19:03:59.610608 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 19:03:59.610646 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":114,"threads_started":1}
I20260925 19:04:00.489850 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:00.489578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:43591)
0925 19:04:00.489755 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:00.489792 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
I20260925 19:04:00.611323 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:00.611115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:47915)
0925 19:04:00.611242 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:00.611278 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260925 19:04:00.611340 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:00.611116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:55223)
0925 19:04:00.611247 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:00.611279 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260925 19:04:01.490653 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:01.490427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:33483)
0925 19:04:01.490563 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:01.490607 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260925 19:04:01.611985 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:01.611795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:35245)
0925 19:04:01.611906 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:01.611938 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260925 19:04:01.612008 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:01.611780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:37791)
0925 19:04:01.611937 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:01.611969 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260925 19:04:02.491302 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:02.491072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:49039)
0925 19:04:02.491224 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:02.491256 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260925 19:04:02.612720 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:02.612497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:38635)
0925 19:04:02.612626 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:02.612662 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260925 19:04:02.612748 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:02.612499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:35053)
0925 19:04:02.612676 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:02.612708 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260925 19:04:03.491998 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:03.491753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:51845)
0925 19:04:03.491889 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:03.491942 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":54,"threads_started":1}
I20260925 19:04:03.613389 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:03.613194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:44755)
0925 19:04:03.613315 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:03.613348 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260925 19:04:03.613389 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:03.613204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:46013)
0925 19:04:03.613318 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:03.613345 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260925 19:04:04.492703 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:04.492505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:42551)
0925 19:04:04.492627 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:04.492658 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260925 19:04:04.614034 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:04.613787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:56441)
0925 19:04:04.613935 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:04.613974 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260925 19:04:04.614243 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:04.613917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:36361)
0925 19:04:04.614050 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:04.614084 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260925 19:04:05.493433 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:05.493173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:42693)
0925 19:04:05.493349 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:05.493387 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":75,"threads_started":1}
I20260925 19:04:05.614873 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:05.614670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:40429)
0925 19:04:05.614791 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:05.614828 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260925 19:04:05.614897 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:05.614650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:46781)
0925 19:04:05.614824 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:05.614857 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260925 19:04:06.494237 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:06.494035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.1:58347)
0925 19:04:06.494158 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:06.494195 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260925 19:04:06.615562 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:06.615317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.2:35377)
0925 19:04:06.615472 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:06.615504 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260925 19:04:06.615577 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:06.615328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37199 (local address 127.0.98.3:34959)
0925 19:04:06.615501 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:06.615540 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
I20260925 19:04:06.790125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 16877
I20260925 19:04:06.794878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 17045
I20260925 19:04:06.801268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 17181
2026-09-25T19:04:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58138 ms)
[----------] 30 tests from SecurityITest (252762 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:06 dist-test-slave-49bz krb5kdc[17378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:06 dist-test-slave-49bz krb5kdc[17378](info): set up 2 sockets
Sep 25 19:04:06 dist-test-slave-49bz krb5kdc[17378](info): commencing operation
krb5kdc: starting...
W20260925 19:04:07.814095 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:07 dist-test-slave-49bz krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:07.926414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:08.000104 17391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:08.000339 17391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:08.000373 17391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:08.001415 17391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:08.001487 17391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:08.001508 17391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:08.001528 17391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:08.001554 17391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:08.003052 17391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:08.003324 17391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:08.003551 17391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:08.009101 17397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:08.009171 17396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:08.009183 17391 server_base.cc:1034] running on GCE node
W20260925 19:04:08.009291 17399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:08.009569 17391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:08.009871 17391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:08.011044 17391 hybrid_clock.cc:648] HybridClock initialized: now 1790363048011037 us; error 34 us; skew 500 ppm
Sep 25 19:04:08 dist-test-slave-49bz krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:08.013368 17391 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:08.013738 17391 webserver.cc:460] Webserver started at http://127.0.98.62:42209/ using document root <none> and password file <none>
I20260925 19:04:08.013965 17391 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:08.014020 17391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:08.014145 17391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:08.015182 17391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "08dc9f0cf9f34abebed652c0c6ad4902"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "90f768d9f0020fa78d887c780d7469d1"
server_key_iv: "8395d9fc82cd19dccebf2023ae3faf10"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.015517 17391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "08dc9f0cf9f34abebed652c0c6ad4902"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "90f768d9f0020fa78d887c780d7469d1"
server_key_iv: "8395d9fc82cd19dccebf2023ae3faf10"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.016712 17391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:08.017354 17406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.017532 17391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:08.017599 17391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "08dc9f0cf9f34abebed652c0c6ad4902"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "90f768d9f0020fa78d887c780d7469d1"
server_key_iv: "8395d9fc82cd19dccebf2023ae3faf10"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.017681 17391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:08.029374 17391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:08.031242 17391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:08.031391 17391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:08.035185 17391 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35517
I20260925 19:04:08.035234 17458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35517 every 8 connection(s)
I20260925 19:04:08.035566 17391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:08.036331 17459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:08.038364 17459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902: Bootstrap starting.
I20260925 19:04:08.038887 17459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:08.039148 17459 log.cc:824] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:08.039767 17459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902: No bootstrap required, opened a new log
I20260925 19:04:08.040412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 17391
I20260925 19:04:08.040527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:08.040709 392 external_mini_cluster.cc:1428] Setting key badd42f3da28258da7a25652275e43fb
I20260925 19:04:08.041879 17459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc9f0cf9f34abebed652c0c6ad4902" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } }
I20260925 19:04:08.042045 17459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:08.042083 17459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08dc9f0cf9f34abebed652c0c6ad4902, State: Initialized, Role: FOLLOWER
I20260925 19:04:08.042232 17459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc9f0cf9f34abebed652c0c6ad4902" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } }
I20260925 19:04:08.042322 17459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:08.042346 17459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:08.042415 17459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:08.043427 17459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc9f0cf9f34abebed652c0c6ad4902" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } }
I20260925 19:04:08.043529 17459 leader_election.cc:302] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08dc9f0cf9f34abebed652c0c6ad4902; no voters:
I20260925 19:04:08.043676 17459 leader_election.cc:288] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:08.043715 17464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:08.043905 17459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:08.043912 17464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [term 1 LEADER]: Becoming Leader. State: Replica: 08dc9f0cf9f34abebed652c0c6ad4902, State: Running, Role: LEADER
I20260925 19:04:08.044155 17464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc9f0cf9f34abebed652c0c6ad4902" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } }
I20260925 19:04:08.044495 17465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08dc9f0cf9f34abebed652c0c6ad4902. Latest consensus state: current_term: 1 leader_uuid: "08dc9f0cf9f34abebed652c0c6ad4902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc9f0cf9f34abebed652c0c6ad4902" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } } }
I20260925 19:04:08.044502 17466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08dc9f0cf9f34abebed652c0c6ad4902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc9f0cf9f34abebed652c0c6ad4902" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } } }
I20260925 19:04:08.044711 17466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:08.044919 17465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:08.045135 17473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:08.045591 17473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:08.047174 17473 catalog_manager.cc:1347] Generated new cluster ID: d0b864f7fe484b77a18bcce923a85c68
I20260925 19:04:08.047227 17473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:04:08 dist-test-slave-49bz krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:08.052115 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.041227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.0.98.62:35517)
0925 19:04:08.041424 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.041433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.041468 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:04:08.041844 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.041852 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.041912 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.042055 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:08.042849 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.044258 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.045291 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.045829 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.045941 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.048605 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:08.048609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:08.048609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:08.048619 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:08.050496 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:08.050828 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:08.050831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:08.050831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:08.050948 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:08.051113 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:08.051114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:08.051114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:08.051361 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:08.051491 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.051832 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.051944 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260925 19:04:08.054018 17473 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:08.054576 17473 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:08.064254 17473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902: Generated new TSK 0
I20260925 19:04:08.064445 17473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:04:08.087169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35517
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:08.163638 17485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:08.163847 17485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:08.163869 17485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:08.164949 17485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:08.164985 17485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:08.165030 17485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:04:08.166343 17485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:08.166563 17485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:08.166811 17485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:08.171365 17490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:08.171373 17491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:08.171370 17493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:08.171629 17485 server_base.cc:1034] running on GCE node
I20260925 19:04:08.171847 17485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:08.172081 17485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:08.173246 17485 hybrid_clock.cc:648] HybridClock initialized: now 1790363048173231 us; error 25 us; skew 500 ppm
Sep 25 19:04:08 dist-test-slave-49bz krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:08.175645 17485 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:04:08.176046 17485 webserver.cc:460] Webserver started at http://127.0.98.1:41555/ using document root <none> and password file <none>
I20260925 19:04:08.176292 17485 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:08.176384 17485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:08.176523 17485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:08.177500 17485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "2d10f53b1a554815a66325555830149b"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "7b0139b6c8b58557ca69c91e87140ff9"
server_key_iv: "e9dad1c2a016b3704870c3afc14a72a8"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.177831 17485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2d10f53b1a554815a66325555830149b"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "7b0139b6c8b58557ca69c91e87140ff9"
server_key_iv: "e9dad1c2a016b3704870c3afc14a72a8"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.179121 17485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:08.179800 17500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.179975 17485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.180075 17485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "2d10f53b1a554815a66325555830149b"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "7b0139b6c8b58557ca69c91e87140ff9"
server_key_iv: "e9dad1c2a016b3704870c3afc14a72a8"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.180145 17485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:08.190128 17485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:08.191928 17485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:08.192073 17485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:08.192278 17485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:08.192584 17485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:08.192636 17485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.192672 17485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:08.192718 17485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.198319 17485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44645
I20260925 19:04:08.198361 17613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44645 every 8 connection(s)
I20260925 19:04:08.198673 17485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:04:08.201330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 17485
I20260925 19:04:08.201442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:04:08.201627 392 external_mini_cluster.cc:1428] Setting key 512b139ce29faf7de043e334ad3e25d3
Sep 25 19:04:08 dist-test-slave-49bz krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:08.208812 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.199397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34079 (local address 127.0.98.62:35517)
0925 19:04:08.199442 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.199444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.199612 (+ 168us) server_negotiation.cc:407] Connection header received
0925 19:04:08.200480 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.200481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.200492 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.200543 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:08.201148 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.201718 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.203222 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.203424 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.203479 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.205829 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:08.205831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:08.205831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:08.205843 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:08.207537 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:08.207914 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:08.207915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:08.207916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:08.207975 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:08.208203 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:08.208204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:08.208204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:08.208334 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:08.208403 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.208616 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.208700 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:04:08.208828 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.199355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35517 (local address 127.0.98.1:34079)
0925 19:04:08.199507 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:08.199546 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:08.200398 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:08.200619 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:08.200623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:08.200903 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:08.201076 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.201080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.201809 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.201810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.203136 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.203139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.203509 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.203510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.203668 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.204283 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:08.204293 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:08.205717 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:08.207616 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:08.207622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:08.207623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:08.207839 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:08.208042 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:08.208042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:08.208042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:08.208144 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:08.208394 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:08.208398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:08.208497 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 19:04:08.208570 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 19:04:08.208673 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260925 19:04:08.209393 17614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35517
I20260925 19:04:08.209468 17614 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:08.209645 17614 heartbeater.cc:511] Master 127.0.98.62:35517 requested a full tablet report, sending...
I20260925 19:04:08.210176 17423 ts_manager.cc:194] Registered new tserver with Master: 2d10f53b1a554815a66325555830149b (127.0.98.1:44645)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 19:04:08.210873 17423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:04:08.229130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35517
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:08.306054 17619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:08.306238 17619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:08.306344 17619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:08.307483 17619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:08.307518 17619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:08.307544 17619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:04:08.308900 17619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:08.309072 17619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:08.309229 17619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:08.313719 17627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:08.313712 17624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:08.313988 17619 server_base.cc:1034] running on GCE node
W20260925 19:04:08.313699 17625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:08.314322 17619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:08.314543 17619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:08.315671 17619 hybrid_clock.cc:648] HybridClock initialized: now 1790363048315649 us; error 37 us; skew 500 ppm
Sep 25 19:04:08 dist-test-slave-49bz krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:08.317965 17619 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:04:08.318434 17619 webserver.cc:460] Webserver started at http://127.0.98.2:36787/ using document root <none> and password file <none>
I20260925 19:04:08.318601 17619 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:08.318636 17619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:08.318715 17619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:08.319577 17619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "2271650bfd9c430788d56a4dc3b878e3"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "482117a553dfdad45dfccb703a141c3f"
server_key_iv: "6b74f6bc66859a0a166c3efc37ea34b2"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.319970 17619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2271650bfd9c430788d56a4dc3b878e3"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "482117a553dfdad45dfccb703a141c3f"
server_key_iv: "6b74f6bc66859a0a166c3efc37ea34b2"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.321188 17619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:08.321792 17634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.321939 17619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.322019 17619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "2271650bfd9c430788d56a4dc3b878e3"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "482117a553dfdad45dfccb703a141c3f"
server_key_iv: "6b74f6bc66859a0a166c3efc37ea34b2"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.322069 17619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:08.347966 17619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:08.349792 17619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:08.349912 17619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:08.350085 17619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:08.350453 17619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:08.350494 17619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.350538 17619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:08.350565 17619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.356041 17619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45235
I20260925 19:04:08.356122 17747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45235 every 8 connection(s)
I20260925 19:04:08.356428 17619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 19:04:08 dist-test-slave-49bz krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:08.363737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 17619
I20260925 19:04:08.363857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:04:08.364063 392 external_mini_cluster.cc:1428] Setting key 620b3d8f79f5f0fe77d6e15a103e3615
I20260925 19:04:08.365458 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.357188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43611 (local address 127.0.98.62:35517)
0925 19:04:08.357248 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.357250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.357421 (+ 171us) server_negotiation.cc:407] Connection header received
0925 19:04:08.358305 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.358305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.358317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.358377 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:08.358978 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.359544 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.360465 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.360637 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.360684 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.362607 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:08.362608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:08.362609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:08.362614 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:08.364105 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:08.364494 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:08.364495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:08.364496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:08.364553 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:08.364861 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:08.364862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:08.364863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:08.364969 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:08.365017 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.365336 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.365398 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:04:08.365744 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.357135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35517 (local address 127.0.98.2:43611)
0925 19:04:08.357295 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:08.357336 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:08.358198 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:08.358449 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:08.358453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:08.358688 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:08.358909 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.358913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.359620 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.359621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.360392 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.360394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.360702 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.360702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.360806 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.361221 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:08.361228 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:08.362530 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:08.364188 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:08.364195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:08.364215 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:08.364431 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:08.364638 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:08.364639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:08.364639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:08.364786 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:08.365032 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:08.365037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:08.365184 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 19:04:08.365313 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 19:04:08.365482 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260925 19:04:08.366690 17748 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35517
I20260925 19:04:08.366770 17748 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:08.366997 17748 heartbeater.cc:511] Master 127.0.98.62:35517 requested a full tablet report, sending...
I20260925 19:04:08.367383 17423 ts_manager.cc:194] Registered new tserver with Master: 2271650bfd9c430788d56a4dc3b878e3 (127.0.98.2:45235)
I20260925 19:04:08.367754 17423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43611
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:04:08.388464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35517
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:08.464900 17753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:08.465063 17753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:08.465082 17753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:08.466091 17753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:08.466126 17753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:08.466148 17753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:04:08.467525 17753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:08.467748 17753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:08.467988 17753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:04:08.472659 17753 server_base.cc:1034] running on GCE node
W20260925 19:04:08.472702 17759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:08.472642 17758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:08.472683 17761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:08.473062 17753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:08.473321 17753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:08.474457 17753 hybrid_clock.cc:648] HybridClock initialized: now 1790363048474433 us; error 42 us; skew 500 ppm
Sep 25 19:04:08 dist-test-slave-49bz krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:08.476823 17753 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:04:08.477257 17753 webserver.cc:460] Webserver started at http://127.0.98.3:42233/ using document root <none> and password file <none>
I20260925 19:04:08.477444 17753 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:08.477483 17753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:08.477562 17753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:08.478523 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "a8df8f813bc746b3aef7d88af2e19491"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "fc7b2a6fe40ec4088e720432e143ca7c"
server_key_iv: "ba5d14390df78ec268de09c876a3f9be"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.478797 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a8df8f813bc746b3aef7d88af2e19491"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "fc7b2a6fe40ec4088e720432e143ca7c"
server_key_iv: "ba5d14390df78ec268de09c876a3f9be"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.479974 17753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:08.480631 17768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.480834 17753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:08.480901 17753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "a8df8f813bc746b3aef7d88af2e19491"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "fc7b2a6fe40ec4088e720432e143ca7c"
server_key_iv: "ba5d14390df78ec268de09c876a3f9be"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.480988 17753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:08.488516 17753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:08.490245 17753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:08.490453 17753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:08.490653 17753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:08.490947 17753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:08.490999 17753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.491037 17753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:08.491063 17753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.496690 17753 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36609
I20260925 19:04:08.496737 17881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36609 every 8 connection(s)
I20260925 19:04:08.496997 17753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:04:08.502592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 17753
Sep 25 19:04:08 dist-test-slave-49bz krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:08.502713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:04:08.502931 392 external_mini_cluster.cc:1428] Setting key d6510045ce24ee22a4582e18cb69e056
I20260925 19:04:08.505898 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.497669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49987 (local address 127.0.98.62:35517)
0925 19:04:08.497751 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.497753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.497958 (+ 205us) server_negotiation.cc:407] Connection header received
0925 19:04:08.498920 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.498920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.498930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.498986 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:08.499547 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.500044 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.500932 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.501091 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.501135 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.503148 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:08.503149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:08.503150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:08.503157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:08.504586 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:08.504983 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:08.504984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:08.504985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:08.505039 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:08.505291 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:08.505292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:08.505292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:08.505403 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:08.505476 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.505727 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.505816 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 19:04:08.506059 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.497653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35517 (local address 127.0.98.3:49987)
0925 19:04:08.497827 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:08.497865 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:08.498828 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:08.499037 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:08.499040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:08.499298 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:08.499478 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.499481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.500129 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.500130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.500862 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.500864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.501154 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.501155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.501239 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.501631 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:08.501638 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:08.503058 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:08.504684 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:08.504691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:08.504693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:08.504914 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:08.505120 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:08.505121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:08.505121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:08.505227 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:08.505468 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:08.505474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:08.505603 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 19:04:08.505704 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 19:04:08.505854 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260925 19:04:08.506461 17882 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35517
I20260925 19:04:08.506532 17882 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:08.506651 17882 heartbeater.cc:511] Master 127.0.98.62:35517 requested a full tablet report, sending...
I20260925 19:04:08.506917 17423 ts_manager.cc:194] Registered new tserver with Master: a8df8f813bc746b3aef7d88af2e19491 (127.0.98.3:36609)
I20260925 19:04:08.507252 17423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49987
I20260925 19:04:08.514521 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:04:08.521685 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.515765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.0.98.62:35517)
0925 19:04:08.515842 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.515844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.515947 (+ 103us) server_negotiation.cc:407] Connection header received
0925 19:04:08.516023 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.516024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.516032 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.516066 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:08.516648 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.517114 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.517866 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.518069 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.518171 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.519349 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:08.519351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:08.519351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:08.519358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:08.520521 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:08.520874 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:08.520875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:08.520876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:08.520928 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:08.521131 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:08.521132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:08.521132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:08.521269 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:08.521368 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.521510 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.521582 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:04:08.523116 17423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:08.524662 17423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:08.530768 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.526732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44645 (local address 127.0.0.1:44058)
0925 19:04:08.527042 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:08.527084 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:08.527152 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:08.527425 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:08.527428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:08.527430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:08.527496 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.527498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.528523 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.528524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.529539 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.529541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.530451 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.530452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.530531 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.530544 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 19:04:08.530618 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:04:08.530666 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":73,"threads_started":1}
I20260925 19:04:08.530763 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.526858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44058 (local address 127.0.98.1:44645)
0925 19:04:08.527241 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.527245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.527265 (+ 20us) server_negotiation.cc:407] Connection header received
0925 19:04:08.527305 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.527307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.527336 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.527421 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:08.527566 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.528437 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.529628 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.530371 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.530440 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.530493 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.530645 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.530684 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":84,"threads_started":1}
I20260925 19:04:08.531083 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.526782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36609 (local address 127.0.0.1:59998)
0925 19:04:08.527050 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:08.527086 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:08.527166 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:08.527749 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:08.527749 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:08.527751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:08.527797 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.527799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.528961 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.528962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.529932 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.529934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.530859 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.530860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.530933 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.530936 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:08.530986 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:04:08.531019 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":9,"thread_start_us":59,"threads_started":1}
I20260925 19:04:08.531155 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.526881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.0.98.3:36609)
0925 19:04:08.527555 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.527560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.527568 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:08.527600 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.527602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.527616 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.527688 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:08.527854 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.528856 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.530008 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.530705 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.530761 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.530823 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.531020 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.531062 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":90,"threads_started":1}
I20260925 19:04:08.531253 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.526944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45235 (local address 127.0.0.1:49204)
0925 19:04:08.527785 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:08.527794 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:08.527848 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:08.528053 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:08.528054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:08.528056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:08.528106 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.528107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.529143 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.529144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.530008 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.530011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.531017 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.531018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.531074 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.531078 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:08.531128 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:04:08.531167 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":65,"threads_started":1}
I20260925 19:04:08.531520 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.527055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.0.98.2:45235)
0925 19:04:08.527230 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.527235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.527904 (+ 669us) server_negotiation.cc:407] Connection header received
0925 19:04:08.527951 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.527953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.527967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.528040 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:08.528158 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.529055 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.530072 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.530846 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.530912 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.530982 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.531399 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.531444 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260925 19:04:08.531767 17548 tablet_service.cc:1467] Processing CreateTablet for tablet 32bac221770f4812ad61b0d3b58d961a (DEFAULT_TABLE table=test-table [id=8d84874199274df6b8d4eb4ef19b3c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:08.531883 17816 tablet_service.cc:1467] Processing CreateTablet for tablet 32bac221770f4812ad61b0d3b58d961a (DEFAULT_TABLE table=test-table [id=8d84874199274df6b8d4eb4ef19b3c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:08.532023 17682 tablet_service.cc:1467] Processing CreateTablet for tablet 32bac221770f4812ad61b0d3b58d961a (DEFAULT_TABLE table=test-table [id=8d84874199274df6b8d4eb4ef19b3c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:08.532156 17548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bac221770f4812ad61b0d3b58d961a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:08.532202 17816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bac221770f4812ad61b0d3b58d961a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:08.532311 17682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bac221770f4812ad61b0d3b58d961a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:08.535015 17896 tablet_bootstrap.cc:492] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b: Bootstrap starting.
I20260925 19:04:08.535320 17898 tablet_bootstrap.cc:492] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: Bootstrap starting.
I20260925 19:04:08.535719 17896 tablet_bootstrap.cc:654] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:08.536008 17898 tablet_bootstrap.cc:654] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:08.536365 17896 log.cc:824] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:08.536603 17898 log.cc:824] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:08.537000 17897 tablet_bootstrap.cc:492] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: Bootstrap starting.
I20260925 19:04:08.537194 17896 tablet_bootstrap.cc:492] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b: No bootstrap required, opened a new log
I20260925 19:04:08.537281 17896 ts_tablet_manager.cc:1397] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:04:08.537405 17898 tablet_bootstrap.cc:492] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: No bootstrap required, opened a new log
I20260925 19:04:08.537472 17898 ts_tablet_manager.cc:1397] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:04:08.537635 17897 tablet_bootstrap.cc:654] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:08.538339 17897 log.cc:824] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:08.539048 17897 tablet_bootstrap.cc:492] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: No bootstrap required, opened a new log
I20260925 19:04:08.539112 17897 ts_tablet_manager.cc:1397] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:04:08.539342 17896 raft_consensus.cc:348] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10f53b1a554815a66325555830149b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44645 } } peers { permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 } } peers { permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 } }
I20260925 19:04:08.539490 17896 raft_consensus.cc:374] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:08.539546 17896 raft_consensus.cc:729] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d10f53b1a554815a66325555830149b, State: Initialized, Role: FOLLOWER
I20260925 19:04:08.539516 17898 raft_consensus.cc:348] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10f53b1a554815a66325555830149b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44645 } } peers { permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 } } peers { permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 } }
I20260925 19:04:08.539680 17898 raft_consensus.cc:374] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:08.539767 17898 raft_consensus.cc:729] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2271650bfd9c430788d56a4dc3b878e3, State: Initialized, Role: FOLLOWER
I20260925 19:04:08.539752 17896 consensus_queue.cc:260] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10f53b1a554815a66325555830149b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44645 } } peers { permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 } } peers { permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 } }
I20260925 19:04:08.539904 17898 consensus_queue.cc:260] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10f53b1a554815a66325555830149b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44645 } } peers { permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 } } peers { permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 } }
I20260925 19:04:08.540019 17896 ts_tablet_manager.cc:1428] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:08.540129 17614 heartbeater.cc:503] Master 127.0.98.62:35517 was elected leader, sending a full tablet report...
I20260925 19:04:08.540159 17898 ts_tablet_manager.cc:1428] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:04:08.540299 17748 heartbeater.cc:503] Master 127.0.98.62:35517 was elected leader, sending a full tablet report...
I20260925 19:04:08.540963 17897 raft_consensus.cc:348] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10f53b1a554815a66325555830149b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44645 } } peers { permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 } } peers { permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 } }
I20260925 19:04:08.541113 17897 raft_consensus.cc:374] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:08.541179 17897 raft_consensus.cc:729] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8df8f813bc746b3aef7d88af2e19491, State: Initialized, Role: FOLLOWER
I20260925 19:04:08.541298 17897 consensus_queue.cc:260] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10f53b1a554815a66325555830149b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44645 } } peers { permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 } } peers { permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 } }
I20260925 19:04:08.541579 17882 heartbeater.cc:503] Master 127.0.98.62:35517 was elected leader, sending a full tablet report...
I20260925 19:04:08.541560 17897 ts_tablet_manager.cc:1428] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260925 19:04:08.606942 17749 tablet.cc:2369] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:04:08.619144 17902 raft_consensus.cc:482] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:08.619252 17902 raft_consensus.cc:504] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10f53b1a554815a66325555830149b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44645 } } peers { permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 } } peers { permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 } }
I20260925 19:04:08.619692 17902 leader_election.cc:288] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2271650bfd9c430788d56a4dc3b878e3 (127.0.98.2:45235), a8df8f813bc746b3aef7d88af2e19491 (127.0.98.3:36609)
I20260925 19:04:08.623353 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.619949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59629 (local address 127.0.98.2:45235)
0925 19:04:08.619988 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.619990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.620078 (+ 88us) server_negotiation.cc:407] Connection header received
0925 19:04:08.620141 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.620142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.620154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.620202 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:08.620341 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.621003 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.622402 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.622989 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.623067 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.623078 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.623243 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.623290 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:04:08.623353 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.619943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53349 (local address 127.0.98.3:36609)
0925 19:04:08.619988 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.619991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.619999 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:08.620024 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.620025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.620033 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.620115 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:08.620302 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.621000 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.622406 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.622992 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.623067 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.623078 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.623243 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.623282 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:04:08.623417 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.619855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36609 (local address 127.0.98.1:53349)
0925 19:04:08.619910 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:08.619921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:08.619960 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:08.620128 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:08.620130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:08.620131 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:08.620217 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.620219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.621077 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.621078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.622320 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.622324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.623074 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.623075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.623158 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.623165 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 19:04:08.623223 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:04:08.623277 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":66}
I20260925 19:04:08.623423 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.619855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45235 (local address 127.0.98.1:59629)
0925 19:04:08.620018 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:08.620025 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:08.620067 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:08.620219 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:08.620220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:08.620221 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:08.620279 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.620280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.621077 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.621078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.622320 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.622324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.623074 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.623075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.623158 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.623166 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 19:04:08.623223 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 19:04:08.623277 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":97,"thread_start_us":92,"threads_started":1}
I20260925 19:04:08.623768 17836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bac221770f4812ad61b0d3b58d961a" candidate_uuid: "2d10f53b1a554815a66325555830149b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8df8f813bc746b3aef7d88af2e19491" is_pre_election: true
I20260925 19:04:08.623885 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bac221770f4812ad61b0d3b58d961a" candidate_uuid: "2d10f53b1a554815a66325555830149b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2271650bfd9c430788d56a4dc3b878e3" is_pre_election: true
I20260925 19:04:08.623929 17836 raft_consensus.cc:2454] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d10f53b1a554815a66325555830149b in term 0.
I20260925 19:04:08.624022 17702 raft_consensus.cc:2454] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d10f53b1a554815a66325555830149b in term 0.
I20260925 19:04:08.624163 17501 leader_election.cc:302] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d10f53b1a554815a66325555830149b, a8df8f813bc746b3aef7d88af2e19491; no voters:
I20260925 19:04:08.624270 17902 raft_consensus.cc:2781] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:08.624320 17902 raft_consensus.cc:482] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:08.624342 17902 raft_consensus.cc:3037] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:08.625080 17902 raft_consensus.cc:504] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10f53b1a554815a66325555830149b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44645 } } peers { permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 } } peers { permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 } }
I20260925 19:04:08.625221 17902 leader_election.cc:288] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [CANDIDATE]: Term 1 election: Requested vote from peers 2271650bfd9c430788d56a4dc3b878e3 (127.0.98.2:45235), a8df8f813bc746b3aef7d88af2e19491 (127.0.98.3:36609)
I20260925 19:04:08.625394 17836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bac221770f4812ad61b0d3b58d961a" candidate_uuid: "2d10f53b1a554815a66325555830149b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8df8f813bc746b3aef7d88af2e19491"
I20260925 19:04:08.625401 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bac221770f4812ad61b0d3b58d961a" candidate_uuid: "2d10f53b1a554815a66325555830149b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2271650bfd9c430788d56a4dc3b878e3"
I20260925 19:04:08.625479 17836 raft_consensus.cc:3037] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:08.625476 17702 raft_consensus.cc:3037] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:08.626142 17702 raft_consensus.cc:2454] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d10f53b1a554815a66325555830149b in term 1.
I20260925 19:04:08.626238 17836 raft_consensus.cc:2454] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d10f53b1a554815a66325555830149b in term 1.
I20260925 19:04:08.626333 17502 leader_election.cc:302] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2271650bfd9c430788d56a4dc3b878e3, 2d10f53b1a554815a66325555830149b; no voters:
I20260925 19:04:08.626434 17902 raft_consensus.cc:2781] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:08.626590 17902 raft_consensus.cc:686] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 1 LEADER]: Becoming Leader. State: Replica: 2d10f53b1a554815a66325555830149b, State: Running, Role: LEADER
I20260925 19:04:08.626690 17902 consensus_queue.cc:237] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10f53b1a554815a66325555830149b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44645 } } peers { permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 } } peers { permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 } }
I20260925 19:04:08.627422 17423 catalog_manager.cc:5495] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d10f53b1a554815a66325555830149b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2d10f53b1a554815a66325555830149b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10f53b1a554815a66325555830149b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20260925 19:04:08.644676 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.642504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.0.98.1:44645)
0925 19:04:08.642562 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.642564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.642571 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:04:08.642602 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.642603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.642621 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.642683 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:08.642802 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.643240 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.644012 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.644182 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.644232 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.644354 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:08.644443 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:08.644500 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.644530 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.644546 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:04:08.646476 17702 raft_consensus.cc:1261] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3 [term 1 FOLLOWER]: Refusing update from remote peer 2d10f53b1a554815a66325555830149b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:08.646497 17836 raft_consensus.cc:1261] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491 [term 1 FOLLOWER]: Refusing update from remote peer 2d10f53b1a554815a66325555830149b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:08.646871 17906 consensus_queue.cc:1035] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2271650bfd9c430788d56a4dc3b878e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:08.646991 17902 consensus_queue.cc:1035] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8df8f813bc746b3aef7d88af2e19491" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:08.649197 17909 mvcc.cc:205] Tried to move back new op lower bound from 7333327047253143552 to 7333327047175495680. Current Snapshot: MvccSnapshot[applied={T|T < 7333327047253143552}]
I20260925 19:04:08.649431 17911 mvcc.cc:205] Tried to move back new op lower bound from 7333327047253143552 to 7333327047175495680. Current Snapshot: MvccSnapshot[applied={T|T < 7333327047253143552}]
I20260925 19:04:08.649473 17910 mvcc.cc:205] Tried to move back new op lower bound from 7333327047253143552 to 7333327047175495680. Current Snapshot: MvccSnapshot[applied={T|T < 7333327047253143552}]
I20260925 19:04:08.652227 17422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:08.652339 17422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:08.653087 17422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08dc9f0cf9f34abebed652c0c6ad4902: Sending DeleteTablet for 3 replicas of tablet 32bac221770f4812ad61b0d3b58d961a
I20260925 19:04:08.653389 17548 tablet_service.cc:1514] Processing DeleteTablet for tablet 32bac221770f4812ad61b0d3b58d961a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44058
I20260925 19:04:08.653488 17682 tablet_service.cc:1514] Processing DeleteTablet for tablet 32bac221770f4812ad61b0d3b58d961a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49204
I20260925 19:04:08.653613 17816 tablet_service.cc:1514] Processing DeleteTablet for tablet 32bac221770f4812ad61b0d3b58d961a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59998
I20260925 19:04:08.653635 17919 tablet_replica.cc:330] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b: stopping tablet replica
I20260925 19:04:08.653726 17919 raft_consensus.cc:2229] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:08.653818 17920 tablet_replica.cc:330] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: stopping tablet replica
I20260925 19:04:08.653827 17919 raft_consensus.cc:2258] T 32bac221770f4812ad61b0d3b58d961a P 2d10f53b1a554815a66325555830149b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:08.653900 17920 raft_consensus.cc:2229] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:08.653928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 17485
I20260925 19:04:08.653986 17920 raft_consensus.cc:2258] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:08.654659 17921 tablet_replica.cc:330] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: stopping tablet replica
I20260925 19:04:08.654740 17921 raft_consensus.cc:2229] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:08.654829 17921 raft_consensus.cc:2258] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:08.654863 17920 ts_tablet_manager.cc:1905] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:08.655174 17921 ts_tablet_manager.cc:1905] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:08.656633 17921 ts_tablet_manager.cc:1918] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:08.656723 17921 log.cc:1196] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/32bac221770f4812ad61b0d3b58d961a
I20260925 19:04:08.656875 17920 ts_tablet_manager.cc:1918] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:08.656944 17920 log.cc:1196] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/32bac221770f4812ad61b0d3b58d961a
I20260925 19:04:08.656993 17921 ts_tablet_manager.cc:1939] T 32bac221770f4812ad61b0d3b58d961a P a8df8f813bc746b3aef7d88af2e19491: Deleting consensus metadata
I20260925 19:04:08.657208 17920 ts_tablet_manager.cc:1939] T 32bac221770f4812ad61b0d3b58d961a P 2271650bfd9c430788d56a4dc3b878e3: Deleting consensus metadata
I20260925 19:04:08.657584 17407 catalog_manager.cc:4841] TS a8df8f813bc746b3aef7d88af2e19491 (127.0.98.3:36609): tablet 32bac221770f4812ad61b0d3b58d961a (table test-table [id=8d84874199274df6b8d4eb4ef19b3c0b]) successfully deleted
I20260925 19:04:08.657565 17408 catalog_manager.cc:4841] TS 2271650bfd9c430788d56a4dc3b878e3 (127.0.98.2:45235): tablet 32bac221770f4812ad61b0d3b58d961a (table test-table [id=8d84874199274df6b8d4eb4ef19b3c0b]) successfully deleted
W20260925 19:04:08.660161 17408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44645 (error 108)
I20260925 19:04:08.660351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 17619
I20260925 19:04:08.660872 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.660447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44645 (local address 127.0.0.1:44062)
0925 19:04:08.660810 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:08.660843 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344}
W20260925 19:04:08.661214 17408 catalog_manager.cc:4584] TS 2d10f53b1a554815a66325555830149b (127.0.98.1:44645): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32bac221770f4812ad61b0d3b58d961a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44645: connect: Connection refused (error 111)
I20260925 19:04:08.664974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 17753
I20260925 19:04:08.669059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 17391
2026-09-25T19:04:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1867 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T19:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:08Z Disabled control of system clock
I20260925 19:04:08.695238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35455
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:04:08.771752 17926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:08.771957 17926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:08.771976 17926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:08.773120 17926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:08.773161 17926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:08.773175 17926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:08.773186 17926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:08.773239 17926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:08.774549 17926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:08.774771 17926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:08.775043 17926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:08.779428 17932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:08.779417 17934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:08.779417 17931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:08.779482 17926 server_base.cc:1034] running on GCE node
I20260925 19:04:08.779908 17926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:08.780195 17926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:08.781323 17926 hybrid_clock.cc:648] HybridClock initialized: now 1790363048781312 us; error 28 us; skew 500 ppm
I20260925 19:04:08.782562 17926 webserver.cc:460] Webserver started at http://127.0.98.62:36057/ using document root <none> and password file <none>
I20260925 19:04:08.782809 17926 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:08.782856 17926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:08.783003 17926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:08.784045 17926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "7fe707c31c4f482b80ec90de241b37c2"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "5753af1dccaf6cd63d81ffa8ec5751cd"
server_key_iv: "32645ccf02aa10af09933836a447c721"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.784394 17926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "7fe707c31c4f482b80ec90de241b37c2"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "5753af1dccaf6cd63d81ffa8ec5751cd"
server_key_iv: "32645ccf02aa10af09933836a447c721"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.785557 17926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:08.786221 17940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.786422 17926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:08.786506 17926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "7fe707c31c4f482b80ec90de241b37c2"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "5753af1dccaf6cd63d81ffa8ec5751cd"
server_key_iv: "32645ccf02aa10af09933836a447c721"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.786576 17926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:08.806718 17926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:08.807046 17926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:08.807188 17926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:08.810954 17926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35455
I20260925 19:04:08.811009 17992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35455 every 8 connection(s)
I20260925 19:04:08.811321 17926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:08.812264 17993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:08.814553 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2: Bootstrap starting.
I20260925 19:04:08.815121 17993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:08.815414 17993 log.cc:824] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:08.816044 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2: No bootstrap required, opened a new log
I20260925 19:04:08.817541 17993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe707c31c4f482b80ec90de241b37c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35455 } }
I20260925 19:04:08.817672 17993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:08.817718 17993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe707c31c4f482b80ec90de241b37c2, State: Initialized, Role: FOLLOWER
I20260925 19:04:08.817862 17993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe707c31c4f482b80ec90de241b37c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35455 } }
I20260925 19:04:08.817945 17993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:08.817996 17993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:08.818042 17993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:08.818595 17993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe707c31c4f482b80ec90de241b37c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35455 } }
I20260925 19:04:08.818702 17993 leader_election.cc:302] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fe707c31c4f482b80ec90de241b37c2; no voters:
I20260925 19:04:08.818866 17993 leader_election.cc:288] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:08.818895 17996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:08.819039 17996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [term 1 LEADER]: Becoming Leader. State: Replica: 7fe707c31c4f482b80ec90de241b37c2, State: Running, Role: LEADER
I20260925 19:04:08.819118 17993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:08.819211 17996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe707c31c4f482b80ec90de241b37c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35455 } }
I20260925 19:04:08.819325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 17926
I20260925 19:04:08.819425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:08.819644 392 external_mini_cluster.cc:1428] Setting key 7d798537e68546fc17abd582c67d7be7
I20260925 19:04:08.819692 17998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fe707c31c4f482b80ec90de241b37c2. Latest consensus state: current_term: 1 leader_uuid: "7fe707c31c4f482b80ec90de241b37c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe707c31c4f482b80ec90de241b37c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35455 } } }
I20260925 19:04:08.819763 17998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:08.820092 17997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fe707c31c4f482b80ec90de241b37c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe707c31c4f482b80ec90de241b37c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35455 } } }
I20260925 19:04:08.820148 18005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:08.820178 17997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:08.820669 18005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:08.822378 18005 catalog_manager.cc:1347] Generated new cluster ID: bb08cdc676b84f5cb9528321bdf7f89a
I20260925 19:04:08.822433 18005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:08.825282 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.820426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.0.98.62:35455)
0925 19:04:08.820693 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.820703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.820741 (+ 38us) server_negotiation.cc:407] Connection header received
0925 19:04:08.820857 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.820864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.820938 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.821127 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:08.821344 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.822522 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.823347 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.823794 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.823923 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.824615 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:08.824618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:08.824619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:08.824807 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:08.824838 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:08.824848 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:08.824930 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.825001 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.825080 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":68,"threads_started":1}
I20260925 19:04:08.830943 18005 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:08.831617 18005 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:08.846118 18005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2: Generated new TSK 0
I20260925 19:04:08.846374 18005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:08.847630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35455
--builtin_ntp_servers=127.0.98.20:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:04:08.925994 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:08.926216 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:08.926249 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:08.927374 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:08.927441 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:08.927487 18017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:04:08.928761 18017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:08.928987 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:08.929243 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:08.934002 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:08.934015 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:08.934175 18017 server_base.cc:1034] running on GCE node
W20260925 19:04:08.934017 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:08.934484 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:08.934724 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:08.935882 18017 hybrid_clock.cc:648] HybridClock initialized: now 1790363048935870 us; error 28 us; skew 500 ppm
I20260925 19:04:08.937029 18017 webserver.cc:460] Webserver started at http://127.0.98.1:36935/ using document root <none> and password file <none>
I20260925 19:04:08.937261 18017 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:08.937325 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:08.937446 18017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:08.938544 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "e8d9851caac84dc39a96b4261c3d3059"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "95f26fa6416af343a7906136168710ba"
server_key_iv: "2ea9962f2d9ac3b768d38489582d07a5"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.938886 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e8d9851caac84dc39a96b4261c3d3059"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "95f26fa6416af343a7906136168710ba"
server_key_iv: "2ea9962f2d9ac3b768d38489582d07a5"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.940227 18017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:08.941022 18031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.941198 18017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:08.941304 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "e8d9851caac84dc39a96b4261c3d3059"
format_stamp: "Formatted at 2026-09-25 19:04:08 on dist-test-slave-49bz"
server_key: "95f26fa6416af343a7906136168710ba"
server_key_iv: "2ea9962f2d9ac3b768d38489582d07a5"
server_key_version: "encryptionkey@0"
I20260925 19:04:08.941386 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:08.953914 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:08.954200 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:08.954396 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:08.954618 18017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:08.954932 18017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:08.954981 18017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.955017 18017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:08.955044 18017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:08.960436 18017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33071
I20260925 19:04:08.960505 18144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33071 every 8 connection(s)
I20260925 19:04:08.960829 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:04:08.962366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 18017
I20260925 19:04:08.962476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:04:08.962684 392 external_mini_cluster.cc:1428] Setting key bfd8458c6b40d9698dba4b1c3cad3a90
I20260925 19:04:08.963531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35455
--builtin_ntp_servers=127.0.98.20:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:04:08.966459 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.961572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47537 (local address 127.0.98.62:35455)
0925 19:04:08.961639 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:08.961642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:08.961861 (+ 219us) server_negotiation.cc:407] Connection header received
0925 19:04:08.962399 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:08.962400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:08.962415 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:08.962487 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:08.963219 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.963833 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.965013 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.965188 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.965240 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.966121 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:08.966123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:08.966123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:08.966131 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:08.966147 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:08.966151 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:08.966214 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:08.966317 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 19:04:08.966347 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:04:08.966564 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:08.961562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35455 (local address 127.0.98.1:47537)
0925 19:04:08.961741 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:08.961774 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:08.962242 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:08.962580 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:08.962584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:08.962594 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:08.963131 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.963135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.963937 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.963938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.964917 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:08.964921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:08.965270 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:08.965271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:08.965427 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:08.965998 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:04:08.965998 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:08.966023 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:04:08.966044 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:08.966214 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:08.966216 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:08.966218 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:08.966316 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 19:04:08.966387 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260925 19:04:08.967131 18145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35455
I20260925 19:04:08.967221 18145 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:08.967389 18145 heartbeater.cc:511] Master 127.0.98.62:35455 requested a full tablet report, sending...
I20260925 19:04:08.967850 17957 ts_manager.cc:194] Registered new tserver with Master: e8d9851caac84dc39a96b4261c3d3059 (127.0.98.1:33071)
I20260925 19:04:08.968375 17957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47537
W20260925 19:04:09.039012 18148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:09.039194 18148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:09.039213 18148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:09.040261 18148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:09.040299 18148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:09.040374 18148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:04:09.041709 18148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:09.041941 18148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:09.042244 18148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:09.047132 18156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:09.047166 18154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:09.047302 18153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:09.047556 18148 server_base.cc:1034] running on GCE node
I20260925 19:04:09.047819 18148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:09.048056 18148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:09.049224 18148 hybrid_clock.cc:648] HybridClock initialized: now 1790363049049217 us; error 38 us; skew 500 ppm
I20260925 19:04:09.050578 18148 webserver.cc:460] Webserver started at http://127.0.98.2:42127/ using document root <none> and password file <none>
I20260925 19:04:09.050829 18148 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:09.050895 18148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:09.051035 18148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:09.052065 18148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "237ce723817b48b3bd2bbce5d6e11627"
format_stamp: "Formatted at 2026-09-25 19:04:09 on dist-test-slave-49bz"
server_key: "47d195d599cafa8e5955afff695cf4bc"
server_key_iv: "9b711395a3727d2664fa6140779cb60f"
server_key_version: "encryptionkey@0"
I20260925 19:04:09.052413 18148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "237ce723817b48b3bd2bbce5d6e11627"
format_stamp: "Formatted at 2026-09-25 19:04:09 on dist-test-slave-49bz"
server_key: "47d195d599cafa8e5955afff695cf4bc"
server_key_iv: "9b711395a3727d2664fa6140779cb60f"
server_key_version: "encryptionkey@0"
I20260925 19:04:09.053727 18148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:09.054471 18162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:09.054638 18148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:09.054730 18148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "237ce723817b48b3bd2bbce5d6e11627"
format_stamp: "Formatted at 2026-09-25 19:04:09 on dist-test-slave-49bz"
server_key: "47d195d599cafa8e5955afff695cf4bc"
server_key_iv: "9b711395a3727d2664fa6140779cb60f"
server_key_version: "encryptionkey@0"
I20260925 19:04:09.054824 18148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:09.075237 18148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:09.075552 18148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:09.075696 18148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:09.075923 18148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:09.076236 18148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:09.076285 18148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:09.076327 18148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:09.076352 18148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:09.081923 18148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38093
I20260925 19:04:09.081965 18275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38093 every 8 connection(s)
I20260925 19:04:09.082329 18148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:04:09.087471 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.083052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38793 (local address 127.0.98.62:35455)
0925 19:04:09.083100 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:09.083102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:09.083357 (+ 255us) server_negotiation.cc:407] Connection header received
0925 19:04:09.083927 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:09.083928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:09.083941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:09.083987 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:09.084637 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.085117 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.086087 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.086366 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.086436 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.087198 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:09.087199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:09.087199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:09.087204 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:09.087218 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:09.087222 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:09.087287 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:09.087370 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 19:04:09.087398 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:04:09.087671 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.083038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35455 (local address 127.0.98.2:38793)
0925 19:04:09.083238 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:09.083271 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:09.083812 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:09.084070 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:09.084073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:09.084081 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:09.084549 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.084553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.085213 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.085214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.085970 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.085972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.086456 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.086457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.086581 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.087057 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:04:09.087057 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:09.087082 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:04:09.087103 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:09.087286 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:09.087291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:09.087293 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:09.087373 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 19:04:09.087464 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260925 19:04:09.088143 18276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35455
I20260925 19:04:09.088263 18276 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:09.088434 18276 heartbeater.cc:511] Master 127.0.98.62:35455 requested a full tablet report, sending...
I20260925 19:04:09.088521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 18148
I20260925 19:04:09.088616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:04:09.088799 17957 ts_manager.cc:194] Registered new tserver with Master: 237ce723817b48b3bd2bbce5d6e11627 (127.0.98.2:38093)
I20260925 19:04:09.088831 392 external_mini_cluster.cc:1428] Setting key 6dfbbfffb3e0d0a4737f85d54376de96
I20260925 19:04:09.089397 17957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38793
I20260925 19:04:09.089905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35455
--builtin_ntp_servers=127.0.98.20:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:04:09.167311 18279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:09.167495 18279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:09.167514 18279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:09.168533 18279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:09.168569 18279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:09.168591 18279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:04:09.169890 18279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:09.170153 18279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:09.170446 18279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:09.175047 18285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:09.175105 18284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:09.175108 18287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:09.175347 18279 server_base.cc:1034] running on GCE node
I20260925 19:04:09.175485 18279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:09.175657 18279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:09.176790 18279 hybrid_clock.cc:648] HybridClock initialized: now 1790363049176777 us; error 29 us; skew 500 ppm
I20260925 19:04:09.177987 18279 webserver.cc:460] Webserver started at http://127.0.98.3:40411/ using document root <none> and password file <none>
I20260925 19:04:09.178223 18279 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:09.178313 18279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:09.178471 18279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:09.179457 18279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "92a53cf22cf04b148aae6db346161d95"
format_stamp: "Formatted at 2026-09-25 19:04:09 on dist-test-slave-49bz"
server_key: "165ca45594304d0b1fb76b8edc1ede11"
server_key_iv: "46cd4c3c3b2f9115051b55f1e7f573f0"
server_key_version: "encryptionkey@0"
I20260925 19:04:09.179780 18279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "92a53cf22cf04b148aae6db346161d95"
format_stamp: "Formatted at 2026-09-25 19:04:09 on dist-test-slave-49bz"
server_key: "165ca45594304d0b1fb76b8edc1ede11"
server_key_iv: "46cd4c3c3b2f9115051b55f1e7f573f0"
server_key_version: "encryptionkey@0"
I20260925 19:04:09.181005 18279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 19:04:09.181649 18293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:09.181849 18279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:09.181946 18279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "92a53cf22cf04b148aae6db346161d95"
format_stamp: "Formatted at 2026-09-25 19:04:09 on dist-test-slave-49bz"
server_key: "165ca45594304d0b1fb76b8edc1ede11"
server_key_iv: "46cd4c3c3b2f9115051b55f1e7f573f0"
server_key_version: "encryptionkey@0"
I20260925 19:04:09.182029 18279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:09.197512 18279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:09.197779 18279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:09.197939 18279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:09.198143 18279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:09.198457 18279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:09.198505 18279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:09.198542 18279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:09.198568 18279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:09.204202 18279 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33903
I20260925 19:04:09.204242 18406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33903 every 8 connection(s)
I20260925 19:04:09.204559 18279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:04:09.209239 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.205323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42989 (local address 127.0.98.62:35455)
0925 19:04:09.205365 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:09.205367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:09.205588 (+ 221us) server_negotiation.cc:407] Connection header received
0925 19:04:09.206025 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:09.206026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:09.206036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:09.206093 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:09.206701 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.207179 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.208061 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.208233 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.208282 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.209054 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:09.209055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:09.209055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:09.209059 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:09.209068 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:09.209071 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:09.209106 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:09.209164 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 19:04:09.209182 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:04:09.209417 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.205280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35455 (local address 127.0.98.3:42989)
0925 19:04:09.205459 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:09.205497 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:09.205948 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:09.206142 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:09.206145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:09.206152 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:09.206636 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.206640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.207255 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.207255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.207990 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.207992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.208300 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.208301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.208401 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.208940 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:04:09.208941 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:09.208967 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:04:09.208987 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:09.209108 (+ 121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:09.209110 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:09.209112 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:09.209156 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 19:04:09.209224 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260925 19:04:09.209877 18407 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35455
I20260925 19:04:09.209964 18407 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:09.210127 18407 heartbeater.cc:511] Master 127.0.98.62:35455 requested a full tablet report, sending...
I20260925 19:04:09.210530 17956 ts_manager.cc:194] Registered new tserver with Master: 92a53cf22cf04b148aae6db346161d95 (127.0.98.3:33903)
I20260925 19:04:09.210882 17956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42989
I20260925 19:04:09.214588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 18279
I20260925 19:04:09.214681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:04:09.214890 392 external_mini_cluster.cc:1428] Setting key 3c768e7fbe1a6721359d41a4f634f43b
I20260925 19:04:09.215758 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:04:09.219816 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.217065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.0.98.62:35455)
0925 19:04:09.217120 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:09.217121 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:09.217215 (+ 94us) server_negotiation.cc:407] Connection header received
0925 19:04:09.217258 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:09.217258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:09.217268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:09.217323 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:09.217454 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.217889 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.218630 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.218819 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.218879 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.219620 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:09.219621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:09.219621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:09.219625 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:09.219637 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:09.219641 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:09.219696 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:09.219737 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 19:04:09.219756 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:04:09.221266 17956 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:09.222818 17956 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:09.228928 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.225038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.0.98.2:38093)
0925 19:04:09.225451 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:09.225456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:09.225466 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:04:09.225505 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:09.225507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:09.225522 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:09.225779 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:09.225845 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.226910 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.228079 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.228559 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.228642 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.228729 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:09.228802 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 19:04:09.228844 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":66,"threads_started":1}
I20260925 19:04:09.228919 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.224849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38093 (local address 127.0.0.1:36836)
0925 19:04:09.224975 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:09.225010 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:09.225063 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:09.225634 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:09.225637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:09.225639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:09.225701 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.225702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.226993 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.226994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.228004 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.228006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.228640 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.228641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.228708 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.228720 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 19:04:09.228788 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 19:04:09.228851 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":20,"thread_start_us":84,"threads_started":1}
I20260925 19:04:09.229125 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.225069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33071 (local address 127.0.0.1:41682)
0925 19:04:09.225170 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:09.225178 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:09.225215 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:09.225679 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:09.225680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:09.225682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:09.225739 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.225741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.226654 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.226655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.227835 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.227837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.228708 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.228709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.228763 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.228768 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 19:04:09.228796 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 19:04:09.229045 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":195,"thread_start_us":70,"threads_started":1}
I20260925 19:04:09.229121 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.225201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.0.98.3:33903)
0925 19:04:09.225418 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:09.225423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:09.225434 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:04:09.225493 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:09.225495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:09.225511 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:09.225592 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:09.225790 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.226920 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.227959 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.228616 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.228681 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.228757 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:09.228965 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 19:04:09.229003 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
I20260925 19:04:09.229231 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.224870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33903 (local address 127.0.0.1:37344)
0925 19:04:09.225007 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:09.225018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:09.225069 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:09.225675 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:09.225676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:09.225677 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:09.225712 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.225713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.227005 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.227006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.227859 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.227862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.228785 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.228785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.228846 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.228849 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:09.228891 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 19:04:09.228927 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260925 19:04:09.229562 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.225297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.0.98.1:33071)
0925 19:04:09.225474 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:09.225478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:09.225486 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:09.225591 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:09.225593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:09.225608 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:09.225679 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:09.225794 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.226576 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.227918 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.228616 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.228838 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.228921 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:09.228944 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 19:04:09.229490 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260925 19:04:09.229722 18210 tablet_service.cc:1467] Processing CreateTablet for tablet bf6b8391fa1249afa35462ed86f287fe (DEFAULT_TABLE table=test-table [id=4eaf744395764a87a5da6db8fd29d815]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:09.230130 18210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf6b8391fa1249afa35462ed86f287fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:09.230260 18079 tablet_service.cc:1467] Processing CreateTablet for tablet bf6b8391fa1249afa35462ed86f287fe (DEFAULT_TABLE table=test-table [id=4eaf744395764a87a5da6db8fd29d815]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:09.230592 18079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf6b8391fa1249afa35462ed86f287fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:09.233583 18421 tablet_bootstrap.cc:492] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: Bootstrap starting.
I20260925 19:04:09.234148 18341 tablet_service.cc:1467] Processing CreateTablet for tablet bf6b8391fa1249afa35462ed86f287fe (DEFAULT_TABLE table=test-table [id=4eaf744395764a87a5da6db8fd29d815]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:09.234244 18421 tablet_bootstrap.cc:654] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:09.234469 18341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf6b8391fa1249afa35462ed86f287fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:09.234450 18422 tablet_bootstrap.cc:492] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059: Bootstrap starting.
I20260925 19:04:09.234606 18421 log.cc:824] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:09.235088 18422 tablet_bootstrap.cc:654] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:09.235394 18421 tablet_bootstrap.cc:492] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: No bootstrap required, opened a new log
I20260925 19:04:09.235385 18422 log.cc:824] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:09.235457 18421 ts_tablet_manager.cc:1397] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:09.236155 18422 tablet_bootstrap.cc:492] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059: No bootstrap required, opened a new log
I20260925 19:04:09.236230 18422 ts_tablet_manager.cc:1397] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:09.237406 18425 tablet_bootstrap.cc:492] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: Bootstrap starting.
I20260925 19:04:09.237974 18425 tablet_bootstrap.cc:654] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:09.237996 18421 raft_consensus.cc:348] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 } } peers { permanent_uuid: "e8d9851caac84dc39a96b4261c3d3059" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33071 } } peers { permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 } }
I20260925 19:04:09.238160 18421 raft_consensus.cc:374] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:09.238198 18421 raft_consensus.cc:729] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237ce723817b48b3bd2bbce5d6e11627, State: Initialized, Role: FOLLOWER
I20260925 19:04:09.238334 18425 log.cc:824] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:09.238371 18421 consensus_queue.cc:260] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 } } peers { permanent_uuid: "e8d9851caac84dc39a96b4261c3d3059" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33071 } } peers { permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 } }
I20260925 19:04:09.238636 18421 ts_tablet_manager.cc:1428] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:09.238688 18276 heartbeater.cc:503] Master 127.0.98.62:35455 was elected leader, sending a full tablet report...
I20260925 19:04:09.238734 18422 raft_consensus.cc:348] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 } } peers { permanent_uuid: "e8d9851caac84dc39a96b4261c3d3059" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33071 } } peers { permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 } }
I20260925 19:04:09.238905 18422 raft_consensus.cc:374] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:09.238947 18422 raft_consensus.cc:729] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9851caac84dc39a96b4261c3d3059, State: Initialized, Role: FOLLOWER
I20260925 19:04:09.239060 18422 consensus_queue.cc:260] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 } } peers { permanent_uuid: "e8d9851caac84dc39a96b4261c3d3059" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33071 } } peers { permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 } }
I20260925 19:04:09.239192 18425 tablet_bootstrap.cc:492] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: No bootstrap required, opened a new log
I20260925 19:04:09.239257 18425 ts_tablet_manager.cc:1397] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:04:09.239300 18422 ts_tablet_manager.cc:1428] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:09.239418 18145 heartbeater.cc:503] Master 127.0.98.62:35455 was elected leader, sending a full tablet report...
I20260925 19:04:09.241482 18425 raft_consensus.cc:348] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 } } peers { permanent_uuid: "e8d9851caac84dc39a96b4261c3d3059" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33071 } } peers { permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 } }
I20260925 19:04:09.241595 18425 raft_consensus.cc:374] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:09.241616 18425 raft_consensus.cc:729] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a53cf22cf04b148aae6db346161d95, State: Initialized, Role: FOLLOWER
I20260925 19:04:09.241737 18425 consensus_queue.cc:260] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 } } peers { permanent_uuid: "e8d9851caac84dc39a96b4261c3d3059" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33071 } } peers { permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 } }
I20260925 19:04:09.241976 18425 ts_tablet_manager.cc:1428] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:04:09.242035 18407 heartbeater.cc:503] Master 127.0.98.62:35455 was elected leader, sending a full tablet report...
W20260925 19:04:09.332844 18277 tablet.cc:2369] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:04:09.334547 18428 raft_consensus.cc:482] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:09.334676 18428 raft_consensus.cc:504] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 } } peers { permanent_uuid: "e8d9851caac84dc39a96b4261c3d3059" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33071 } } peers { permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 } }
I20260925 19:04:09.335017 18428 leader_election.cc:288] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 237ce723817b48b3bd2bbce5d6e11627 (127.0.98.2:38093), 92a53cf22cf04b148aae6db346161d95 (127.0.98.3:33903)
I20260925 19:04:09.337821 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.335201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33903 (local address 127.0.98.1:55199)
0925 19:04:09.335241 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:09.335252 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:09.335324 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:09.335481 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:09.335482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:09.335484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:09.335547 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.335549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.336263 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.336264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.337025 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.337027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.337574 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.337575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.337646 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.337650 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:09.337699 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:04:09.337730 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 19:04:09.337873 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.335278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55199 (local address 127.0.98.3:33903)
0925 19:04:09.335345 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:09.335347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:09.335368 (+ 21us) server_negotiation.cc:407] Connection header received
0925 19:04:09.335402 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:09.335402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:09.335412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:09.335465 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:09.335647 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.336193 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.337111 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.337496 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.337573 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.337583 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:09.337721 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 19:04:09.337764 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:04:09.337899 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.335284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51657 (local address 127.0.98.2:38093)
0925 19:04:09.335341 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:09.335342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:09.335624 (+ 282us) server_negotiation.cc:407] Connection header received
0925 19:04:09.335654 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:09.335655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:09.335663 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:09.335719 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:09.335854 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.336313 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.337200 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.337566 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.337629 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.337639 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:09.337786 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 19:04:09.337834 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:04:09.337932 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.335201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38093 (local address 127.0.98.1:51657)
0925 19:04:09.335536 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:09.335543 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:09.335588 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:09.335726 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:09.335726 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:09.335728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:09.335791 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.335793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.336387 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.336388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.337118 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:09.337120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.337634 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.337634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:09.337707 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.337713 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 19:04:09.337767 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 19:04:09.337800 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":173,"thread_start_us":94,"threads_started":1}
I20260925 19:04:09.338315 18361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf6b8391fa1249afa35462ed86f287fe" candidate_uuid: "e8d9851caac84dc39a96b4261c3d3059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a53cf22cf04b148aae6db346161d95" is_pre_election: true
I20260925 19:04:09.338497 18361 raft_consensus.cc:2454] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9851caac84dc39a96b4261c3d3059 in term 0.
I20260925 19:04:09.338528 18230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf6b8391fa1249afa35462ed86f287fe" candidate_uuid: "e8d9851caac84dc39a96b4261c3d3059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237ce723817b48b3bd2bbce5d6e11627" is_pre_election: true
I20260925 19:04:09.338680 18230 raft_consensus.cc:2454] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9851caac84dc39a96b4261c3d3059 in term 0.
I20260925 19:04:09.338908 18035 leader_election.cc:302] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237ce723817b48b3bd2bbce5d6e11627, e8d9851caac84dc39a96b4261c3d3059; no voters:
I20260925 19:04:09.339041 18428 raft_consensus.cc:2781] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:09.339092 18428 raft_consensus.cc:482] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:09.339135 18428 raft_consensus.cc:3037] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:09.339881 18428 raft_consensus.cc:504] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 } } peers { permanent_uuid: "e8d9851caac84dc39a96b4261c3d3059" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33071 } } peers { permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 } }
I20260925 19:04:09.340018 18428 leader_election.cc:288] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [CANDIDATE]: Term 1 election: Requested vote from peers 237ce723817b48b3bd2bbce5d6e11627 (127.0.98.2:38093), 92a53cf22cf04b148aae6db346161d95 (127.0.98.3:33903)
I20260925 19:04:09.340144 18230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf6b8391fa1249afa35462ed86f287fe" candidate_uuid: "e8d9851caac84dc39a96b4261c3d3059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237ce723817b48b3bd2bbce5d6e11627"
I20260925 19:04:09.340181 18361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf6b8391fa1249afa35462ed86f287fe" candidate_uuid: "e8d9851caac84dc39a96b4261c3d3059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a53cf22cf04b148aae6db346161d95"
I20260925 19:04:09.340219 18230 raft_consensus.cc:3037] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:09.340291 18361 raft_consensus.cc:3037] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:09.340930 18230 raft_consensus.cc:2454] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9851caac84dc39a96b4261c3d3059 in term 1.
I20260925 19:04:09.341010 18361 raft_consensus.cc:2454] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9851caac84dc39a96b4261c3d3059 in term 1.
I20260925 19:04:09.341078 18035 leader_election.cc:302] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237ce723817b48b3bd2bbce5d6e11627, e8d9851caac84dc39a96b4261c3d3059; no voters:
I20260925 19:04:09.341166 18428 raft_consensus.cc:2781] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:09.341284 18428 raft_consensus.cc:686] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [term 1 LEADER]: Becoming Leader. State: Replica: e8d9851caac84dc39a96b4261c3d3059, State: Running, Role: LEADER
I20260925 19:04:09.341349 18428 consensus_queue.cc:237] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 } } peers { permanent_uuid: "e8d9851caac84dc39a96b4261c3d3059" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33071 } } peers { permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 } }
I20260925 19:04:09.342234 17957 catalog_manager.cc:5495] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d9851caac84dc39a96b4261c3d3059 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e8d9851caac84dc39a96b4261c3d3059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d9851caac84dc39a96b4261c3d3059" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 } health_report { overall_health: UNKNOWN } } }
I20260925 19:04:09.375877 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.373712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.0.98.1:33071)
0925 19:04:09.373785 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:09.373787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:09.373795 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:09.373831 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:09.373831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:09.373849 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:09.373910 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:09.374017 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.374478 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.375160 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:09.375345 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:09.375394 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:09.375517 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:09.375613 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:09.375669 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:09.375703 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 19:04:09.375719 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:04:09.377609 18230 raft_consensus.cc:1261] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627 [term 1 FOLLOWER]: Refusing update from remote peer e8d9851caac84dc39a96b4261c3d3059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:09.377609 18361 raft_consensus.cc:1261] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95 [term 1 FOLLOWER]: Refusing update from remote peer e8d9851caac84dc39a96b4261c3d3059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:09.377970 18431 consensus_queue.cc:1035] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a53cf22cf04b148aae6db346161d95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:09.378101 18428 consensus_queue.cc:1035] T bf6b8391fa1249afa35462ed86f287fe P e8d9851caac84dc39a96b4261c3d3059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237ce723817b48b3bd2bbce5d6e11627" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:09.380699 18434 mvcc.cc:205] Tried to move back new op lower bound from 7333327050248093696 to 7333327050102796288. Current Snapshot: MvccSnapshot[applied={T|T < 7333327050248093696}]
I20260925 19:04:09.380863 18435 mvcc.cc:205] Tried to move back new op lower bound from 7333327050248093696 to 7333327050102796288. Current Snapshot: MvccSnapshot[applied={T|T < 7333327050248093696}]
I20260925 19:04:09.383132 17956 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:09.383229 17956 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:09.384037 17956 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fe707c31c4f482b80ec90de241b37c2: Sending DeleteTablet for 3 replicas of tablet bf6b8391fa1249afa35462ed86f287fe
I20260925 19:04:09.384348 18210 tablet_service.cc:1514] Processing DeleteTablet for tablet bf6b8391fa1249afa35462ed86f287fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:09 UTC) from {username='slave'} at 127.0.0.1:36836
I20260925 19:04:09.384400 18341 tablet_service.cc:1514] Processing DeleteTablet for tablet bf6b8391fa1249afa35462ed86f287fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:09 UTC) from {username='slave'} at 127.0.0.1:37344
I20260925 19:04:09.384373 18079 tablet_service.cc:1514] Processing DeleteTablet for tablet bf6b8391fa1249afa35462ed86f287fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:09 UTC) from {username='slave'} at 127.0.0.1:41682
I20260925 19:04:09.384573 18444 tablet_replica.cc:330] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: stopping tablet replica
I20260925 19:04:09.384801 18444 raft_consensus.cc:2229] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:09.384804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 18017
I20260925 19:04:09.385455 18444 raft_consensus.cc:2258] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:09.385663 18445 tablet_replica.cc:330] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: stopping tablet replica
I20260925 19:04:09.385748 18445 raft_consensus.cc:2229] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:09.385839 18445 raft_consensus.cc:2258] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:09.385938 18444 ts_tablet_manager.cc:1905] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:09.386277 18445 ts_tablet_manager.cc:1905] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:09.387741 18444 ts_tablet_manager.cc:1918] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:09.387840 18444 log.cc:1196] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/bf6b8391fa1249afa35462ed86f287fe
I20260925 19:04:09.387861 18445 ts_tablet_manager.cc:1918] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:09.387929 18445 log.cc:1196] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/bf6b8391fa1249afa35462ed86f287fe
I20260925 19:04:09.388128 18444 ts_tablet_manager.cc:1939] T bf6b8391fa1249afa35462ed86f287fe P 237ce723817b48b3bd2bbce5d6e11627: Deleting consensus metadata
I20260925 19:04:09.388181 18445 ts_tablet_manager.cc:1939] T bf6b8391fa1249afa35462ed86f287fe P 92a53cf22cf04b148aae6db346161d95: Deleting consensus metadata
I20260925 19:04:09.388567 17944 catalog_manager.cc:4841] TS 237ce723817b48b3bd2bbce5d6e11627 (127.0.98.2:38093): tablet bf6b8391fa1249afa35462ed86f287fe (table test-table [id=4eaf744395764a87a5da6db8fd29d815]) successfully deleted
I20260925 19:04:09.388652 17941 catalog_manager.cc:4841] TS 92a53cf22cf04b148aae6db346161d95 (127.0.98.3:33903): tablet bf6b8391fa1249afa35462ed86f287fe (table test-table [id=4eaf744395764a87a5da6db8fd29d815]) successfully deleted
W20260925 19:04:09.391033 17944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33071 (error 108)
I20260925 19:04:09.391278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 18148
I20260925 19:04:09.391485 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:09.391373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33071 (local address 127.0.0.1:41696)
0925 19:04:09.391413 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:09.391450 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 19:04:09.391819 17944 catalog_manager.cc:4584] TS e8d9851caac84dc39a96b4261c3d3059 (127.0.98.1:33071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf6b8391fa1249afa35462ed86f287fe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33071: connect: Connection refused (error 111)
I20260925 19:04:09.396471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 18279
I20260925 19:04:09.400918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 17926
2026-09-25T19:04:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (731 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:09 dist-test-slave-49bz krb5kdc[18450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:09 dist-test-slave-49bz krb5kdc[18450](info): set up 2 sockets
Sep 25 19:04:09 dist-test-slave-49bz krb5kdc[18450](info): commencing operation
krb5kdc: starting...
W20260925 19:04:10.414000 392 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:10 dist-test-slave-49bz krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:10.524411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41349
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:10.601925 18463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:10.602118 18463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:10.602139 18463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:10.603344 18463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:10.603387 18463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:10.603402 18463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:10.603415 18463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:10.603469 18463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:10.604908 18463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:10.605158 18463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:10.605392 18463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:10.610013 18469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:10.610086 18468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:10.610162 18463 server_base.cc:1034] running on GCE node
W20260925 19:04:10.610041 18471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:10.610579 18463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:10.610842 18463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:10.611981 18463 hybrid_clock.cc:648] HybridClock initialized: now 1790363050611964 us; error 37 us; skew 500 ppm
Sep 25 19:04:10 dist-test-slave-49bz krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:10.614409 18463 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:10.614801 18463 webserver.cc:460] Webserver started at http://127.0.98.62:39155/ using document root <none> and password file <none>
I20260925 19:04:10.615067 18463 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:10.615130 18463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:10.615283 18463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:10.616344 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "a4cce2cb918a482eaacf3e5750849abd"
format_stamp: "Formatted at 2026-09-25 19:04:10 on dist-test-slave-49bz"
server_key: "eb9c7c077ef1ba707918cb41ad842cd8"
server_key_iv: "12c167bec9b2ab2d38a188ff2738494d"
server_key_version: "encryptionkey@0"
I20260925 19:04:10.616705 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4cce2cb918a482eaacf3e5750849abd"
format_stamp: "Formatted at 2026-09-25 19:04:10 on dist-test-slave-49bz"
server_key: "eb9c7c077ef1ba707918cb41ad842cd8"
server_key_iv: "12c167bec9b2ab2d38a188ff2738494d"
server_key_version: "encryptionkey@0"
I20260925 19:04:10.617985 18463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:10.618781 18478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:10.618956 18463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:10.619053 18463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "a4cce2cb918a482eaacf3e5750849abd"
format_stamp: "Formatted at 2026-09-25 19:04:10 on dist-test-slave-49bz"
server_key: "eb9c7c077ef1ba707918cb41ad842cd8"
server_key_iv: "12c167bec9b2ab2d38a188ff2738494d"
server_key_version: "encryptionkey@0"
I20260925 19:04:10.619141 18463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:10.634671 18463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:10.636904 18463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:10.637063 18463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:10.640775 18463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41349
I20260925 19:04:10.640808 18530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41349 every 8 connection(s)
I20260925 19:04:10.641150 18463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:10.642022 18531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:10.644158 18531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd: Bootstrap starting.
I20260925 19:04:10.644657 18531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:10.644874 18531 log.cc:824] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:10.645491 18531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd: No bootstrap required, opened a new log
I20260925 19:04:10.646972 18531 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cce2cb918a482eaacf3e5750849abd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } }
I20260925 19:04:10.647099 18531 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:10.647119 18531 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cce2cb918a482eaacf3e5750849abd, State: Initialized, Role: FOLLOWER
I20260925 19:04:10.647248 18531 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cce2cb918a482eaacf3e5750849abd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } }
I20260925 19:04:10.647326 18531 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:10.647353 18531 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:10.647380 18531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:10.647809 18531 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cce2cb918a482eaacf3e5750849abd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } }
I20260925 19:04:10.647884 18531 leader_election.cc:302] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4cce2cb918a482eaacf3e5750849abd; no voters:
I20260925 19:04:10.647997 18531 leader_election.cc:288] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:10.648069 18534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:10.648218 18531 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:10.648243 18534 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [term 1 LEADER]: Becoming Leader. State: Replica: a4cce2cb918a482eaacf3e5750849abd, State: Running, Role: LEADER
I20260925 19:04:10.648336 18534 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cce2cb918a482eaacf3e5750849abd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } }
I20260925 19:04:10.648820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 18463
I20260925 19:04:10.648936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:10.648948 18536 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4cce2cb918a482eaacf3e5750849abd. Latest consensus state: current_term: 1 leader_uuid: "a4cce2cb918a482eaacf3e5750849abd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cce2cb918a482eaacf3e5750849abd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } } }
I20260925 19:04:10.649138 18536 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:10.649160 392 external_mini_cluster.cc:1428] Setting key c1b6562d54db905a5332e16b87ae06f2
I20260925 19:04:10.648845 18535 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4cce2cb918a482eaacf3e5750849abd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cce2cb918a482eaacf3e5750849abd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } } }
I20260925 19:04:10.649573 18545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:10.650141 18545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:10.649557 18535 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:10.651683 18545 catalog_manager.cc:1347] Generated new cluster ID: b38477feacc9475e9be51129d930a1e8
I20260925 19:04:10.651732 18545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:04:10 dist-test-slave-49bz krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:10.659607 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:10.649799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.0.98.62:41349)
0925 19:04:10.649971 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:10.649981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:10.650110 (+ 129us) server_negotiation.cc:407] Connection header received
0925 19:04:10.650242 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:10.650249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:10.650329 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:10.650476 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:10.651218 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:10.652278 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:10.653079 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:10.653517 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:10.653631 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:10.655917 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:10.655920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:10.655921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:10.655932 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:10.658053 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:10.658487 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:10.658490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:10.658490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:10.658595 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:10.658802 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:10.658803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:10.658803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:10.659036 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:10.659210 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:10.659285 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 19:04:10.659416 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260925 19:04:10.661597 18545 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:10.662086 18545 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:10.673986 18545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd: Generated new TSK 0
I20260925 19:04:10.674161 18545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:04:10.704833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:10.809381 18557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:10.809571 18557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:10.809593 18557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:10.810724 18557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:10.810765 18557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:10.810825 18557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:04:10.812242 18557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:10.812476 18557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:10.812711 18557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:10.817440 18565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:10.817520 18557 server_base.cc:1034] running on GCE node
W20260925 19:04:10.817448 18562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:10.817451 18563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:10.817889 18557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:10.818127 18557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:10.819278 18557 hybrid_clock.cc:648] HybridClock initialized: now 1790363050819250 us; error 33 us; skew 500 ppm
Sep 25 19:04:10 dist-test-slave-49bz krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:10.821777 18557 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:04:10.822182 18557 webserver.cc:460] Webserver started at http://127.0.98.1:40043/ using document root <none> and password file <none>
I20260925 19:04:10.822398 18557 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:10.822441 18557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:10.822518 18557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:10.823545 18557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "5103fd16afe9411ab8d1a2224589e37b"
format_stamp: "Formatted at 2026-09-25 19:04:10 on dist-test-slave-49bz"
server_key: "e4b4b668d73e91833273bf64d6a04235"
server_key_iv: "2a144008f06efe70fd5785bf241d3747"
server_key_version: "encryptionkey@0"
I20260925 19:04:10.823853 18557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5103fd16afe9411ab8d1a2224589e37b"
format_stamp: "Formatted at 2026-09-25 19:04:10 on dist-test-slave-49bz"
server_key: "e4b4b668d73e91833273bf64d6a04235"
server_key_iv: "2a144008f06efe70fd5785bf241d3747"
server_key_version: "encryptionkey@0"
I20260925 19:04:10.825078 18557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:10.825711 18572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:10.825903 18557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:10.825991 18557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "5103fd16afe9411ab8d1a2224589e37b"
format_stamp: "Formatted at 2026-09-25 19:04:10 on dist-test-slave-49bz"
server_key: "e4b4b668d73e91833273bf64d6a04235"
server_key_iv: "2a144008f06efe70fd5785bf241d3747"
server_key_version: "encryptionkey@0"
I20260925 19:04:10.826061 18557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:10.833285 18557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:10.835523 18557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:10.835680 18557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:10.835881 18557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:10.836195 18557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:10.836251 18557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:10.836297 18557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:10.836325 18557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:10.842183 18557 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34289
I20260925 19:04:10.842227 18685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34289 every 8 connection(s)
I20260925 19:04:10.842578 18557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 19:04:10 dist-test-slave-49bz krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:10.849951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 18557
I20260925 19:04:10.850081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:04:10.850318 392 external_mini_cluster.cc:1428] Setting key ce9e9c42fd14bba91859954efc8a681f
I20260925 19:04:10.852005 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:10.843219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36275 (local address 127.0.98.62:41349)
0925 19:04:10.843264 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:10.843266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:10.843501 (+ 235us) server_negotiation.cc:407] Connection header received
0925 19:04:10.844436 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:10.844437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:10.844452 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:10.844499 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:10.845122 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:10.845611 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:10.846590 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:10.846781 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:10.846869 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:10.848997 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:10.848998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:10.848999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:10.849005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:10.850660 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:10.851097 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:10.851097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:10.851097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:10.851149 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:10.851408 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:10.851409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:10.851409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:10.851503 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:10.851551 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:10.851868 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 19:04:10.851947 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:04:10.852294 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:10.843207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41349 (local address 127.0.98.1:36275)
0925 19:04:10.843383 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:10.843415 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:10.844322 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:10.844577 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:10.844581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:10.844826 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:10.845027 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:10.845031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:10.845702 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:10.845703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:10.846501 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:10.846503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:10.846891 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:10.846892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:10.847007 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:10.847439 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:10.847446 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:10.848899 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:10.850732 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:10.850737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:10.850739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:10.851022 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:10.851213 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:10.851214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:10.851214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:10.851348 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:10.851562 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:10.851567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:10.851727 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 19:04:10.851834 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 19:04:10.852031 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260925 19:04:10.852757 18686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41349
I20260925 19:04:10.852849 18686 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:10.853036 18686 heartbeater.cc:511] Master 127.0.98.62:41349 requested a full tablet report, sending...
I20260925 19:04:10.853528 18495 ts_manager.cc:194] Registered new tserver with Master: 5103fd16afe9411ab8d1a2224589e37b (127.0.98.1:34289)
I20260925 19:04:10.854146 18495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36275
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:04:10.876495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:10.955773 18691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:10.955999 18691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:10.956020 18691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:10.957100 18691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:10.957132 18691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:10.957157 18691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:04:10.958618 18691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:10.958808 18691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:10.958971 18691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:10.963568 18697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:10.963551 18696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:10.963546 18699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:10.963827 18691 server_base.cc:1034] running on GCE node
I20260925 19:04:10.964082 18691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:10.964331 18691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:10.965507 18691 hybrid_clock.cc:648] HybridClock initialized: now 1790363050965483 us; error 39 us; skew 500 ppm
Sep 25 19:04:10 dist-test-slave-49bz krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:10.968164 18691 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:04:10.968595 18691 webserver.cc:460] Webserver started at http://127.0.98.2:44309/ using document root <none> and password file <none>
I20260925 19:04:10.968803 18691 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:10.968865 18691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:10.969002 18691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:10.969990 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "b27e788b8b4b441a87cb52263c88a889"
format_stamp: "Formatted at 2026-09-25 19:04:10 on dist-test-slave-49bz"
server_key: "d3d8b29d97569deb0084db0e8ce00363"
server_key_iv: "9ae85c5031f1ed90c18e5b73704a2e94"
server_key_version: "encryptionkey@0"
I20260925 19:04:10.970403 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b27e788b8b4b441a87cb52263c88a889"
format_stamp: "Formatted at 2026-09-25 19:04:10 on dist-test-slave-49bz"
server_key: "d3d8b29d97569deb0084db0e8ce00363"
server_key_iv: "9ae85c5031f1ed90c18e5b73704a2e94"
server_key_version: "encryptionkey@0"
I20260925 19:04:10.971683 18691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:10.972455 18706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:10.972635 18691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:10.972766 18691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "b27e788b8b4b441a87cb52263c88a889"
format_stamp: "Formatted at 2026-09-25 19:04:10 on dist-test-slave-49bz"
server_key: "d3d8b29d97569deb0084db0e8ce00363"
server_key_iv: "9ae85c5031f1ed90c18e5b73704a2e94"
server_key_version: "encryptionkey@0"
I20260925 19:04:10.972848 18691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:10.981643 18691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:10.983940 18691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:10.984100 18691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:10.984328 18691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:10.984634 18691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:10.984689 18691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:10.984730 18691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:10.984759 18691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:10.990363 18691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43945
I20260925 19:04:10.990438 18819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43945 every 8 connection(s)
I20260925 19:04:10.990759 18691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:04:10.990962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 18691
I20260925 19:04:10.991092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:04:10.991382 392 external_mini_cluster.cc:1428] Setting key f9f298b7bd7cb7c12aaef124a6ca2949
Sep 25 19:04:10 dist-test-slave-49bz krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 19:04:11.001080 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:10.991803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59821 (local address 127.0.98.62:41349)
0925 19:04:10.991870 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:10.991872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:10.991882 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:04:10.992878 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:10.992879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:10.992891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:10.992955 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:10.993594 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:10.994100 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:10.995572 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:10.995824 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:10.995880 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:10.998208 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:10.998209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:10.998210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:10.998217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:10.999717 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:11.000112 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:11.000113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:11.000114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:11.000188 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:11.000423 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:11.000424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:11.000424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:11.000534 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:11.000603 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.000912 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.000994 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:04:11.001245 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:10.991541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41349 (local address 127.0.98.2:59821)
0925 19:04:10.991715 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:10.991765 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:10.992733 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:10.993050 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:10.993054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:10.993301 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:10.993482 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:10.993486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:10.994189 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:10.994190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:10.995480 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:10.995484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:10.995911 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:10.995913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:10.996050 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:10.996525 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:10.996536 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:10.998111 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:10.999808 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:10.999815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:10.999823 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:11.000035 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:11.000251 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:11.000251 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:11.000252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:11.000358 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:11.000606 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:11.000613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:11.000766 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 19:04:11.000882 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 19:04:11.001028 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260925 19:04:11.001755 18820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41349
I20260925 19:04:11.001856 18820 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:11.001998 18820 heartbeater.cc:511] Master 127.0.98.62:41349 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 19:04:11.002445 18495 ts_manager.cc:194] Registered new tserver with Master: b27e788b8b4b441a87cb52263c88a889 (127.0.98.2:43945)
I20260925 19:04:11.002892 18495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:04:11.016423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:11.096692 18825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:11.096894 18825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:11.096917 18825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:11.098104 18825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:11.098145 18825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:11.098198 18825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:04:11.099723 18825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:11.099967 18825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:11.100189 18825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:11.104905 18833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:11.104975 18831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:11.104976 18830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:11.105206 18825 server_base.cc:1034] running on GCE node
I20260925 19:04:11.105356 18825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:11.105600 18825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:11.106732 18825 hybrid_clock.cc:648] HybridClock initialized: now 1790363051106722 us; error 30 us; skew 500 ppm
Sep 25 19:04:11 dist-test-slave-49bz krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:11.109082 18825 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:04:11.109540 18825 webserver.cc:460] Webserver started at http://127.0.98.3:38989/ using document root <none> and password file <none>
I20260925 19:04:11.109774 18825 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:11.109822 18825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:11.109954 18825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:11.111017 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "02138dac66ef434e87a5c6b420e55e98"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "84339f621908c9748f8e1394ed7a5eb1"
server_key_iv: "98509451f97559abd1e6a0e289c70381"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.111374 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02138dac66ef434e87a5c6b420e55e98"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "84339f621908c9748f8e1394ed7a5eb1"
server_key_iv: "98509451f97559abd1e6a0e289c70381"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.112622 18825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:11.113341 18840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.113531 18825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:11.113610 18825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "02138dac66ef434e87a5c6b420e55e98"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "84339f621908c9748f8e1394ed7a5eb1"
server_key_iv: "98509451f97559abd1e6a0e289c70381"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.113660 18825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:11.123009 18825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:11.125237 18825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:11.125348 18825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:11.125510 18825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:11.125761 18825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:11.125790 18825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.125811 18825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:11.125825 18825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.131384 18825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33557
I20260925 19:04:11.131423 18953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33557 every 8 connection(s)
I20260925 19:04:11.131724 18825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:04:11 dist-test-slave-49bz krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:11.140820 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.132425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57709 (local address 127.0.98.62:41349)
0925 19:04:11.132488 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.132490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.132724 (+ 234us) server_negotiation.cc:407] Connection header received
0925 19:04:11.133669 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.133670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.133681 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.133733 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:11.134375 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.134851 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.135811 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.135999 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.136085 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.138179 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:11.138180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:11.138181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:11.138188 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:11.139495 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:11.139882 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:11.139883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:11.139884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:11.139945 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:11.140190 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:11.140190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:11.140191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:11.140305 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:11.140385 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.140639 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.140729 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:04:11.140811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 18825
I20260925 19:04:11.140939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:04:11.141032 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.132402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41349 (local address 127.0.98.3:57709)
0925 19:04:11.132577 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:11.132611 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:11.133559 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:11.133824 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:11.133827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:11.134065 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:11.134238 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.134242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.134944 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.134945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.135708 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.135710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.136109 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.136110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.136253 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.136666 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:11.136674 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:11.138084 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:11.139578 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:11.139585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:11.139587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:11.139789 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:11.140017 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:11.140018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:11.140018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:11.140118 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:11.140376 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:11.140382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:11.140513 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 19:04:11.140619 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 19:04:11.140786 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260925 19:04:11.141191 392 external_mini_cluster.cc:1428] Setting key ae19b5483322e35ea5a439bec750749b
I20260925 19:04:11.141592 18954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41349
I20260925 19:04:11.141703 18954 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:11.141880 18954 heartbeater.cc:511] Master 127.0.98.62:41349 requested a full tablet report, sending...
I20260925 19:04:11.142311 18495 ts_manager.cc:194] Registered new tserver with Master: 02138dac66ef434e87a5c6b420e55e98 (127.0.98.3:33557)
I20260925 19:04:11.142701 18495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57709
I20260925 19:04:11.152695 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:04:11.159890 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.154122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.0.98.62:41349)
0925 19:04:11.154162 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.154164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.154289 (+ 125us) server_negotiation.cc:407] Connection header received
0925 19:04:11.154352 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.154353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.154368 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.154415 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:11.154968 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.155418 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.156086 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.156250 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.156297 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.157462 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:11.157463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:11.157463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:11.157469 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:11.158869 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:11.159197 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:11.159198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:11.159198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:11.159246 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:11.159415 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:11.159416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:11.159416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:11.159517 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:11.159582 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.159769 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.159832 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:04:11.161262 18495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:11.162802 18495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:11.168957 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.164935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34289 (local address 127.0.0.1:49472)
0925 19:04:11.165086 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:11.165097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:11.165152 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:11.165427 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:11.165430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:11.165432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:11.165500 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.165501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.166678 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.166679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.167956 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.167959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.168626 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.168626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.168708 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.168722 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 19:04:11.168805 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:04:11.168865 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":7,"thread_start_us":51,"threads_started":1}
I20260925 19:04:11.169077 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.165202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.0.98.2:43945)
0925 19:04:11.165382 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.165387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.165397 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:04:11.165435 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.165437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.165452 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.165536 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:11.165624 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.166730 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.168116 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.168729 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.168788 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.168831 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.168965 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.169004 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20260925 19:04:11.169170 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.165468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.0.98.3:33557)
0925 19:04:11.165774 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.165778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.165786 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:11.165831 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.165833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.165846 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.165921 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:11.166017 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.166981 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.168042 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.168672 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.168740 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.168795 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.169050 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.169090 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":76,"threads_started":1}
I20260925 19:04:11.169196 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.164861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43945 (local address 127.0.0.1:50872)
0925 19:04:11.165048 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:11.165088 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:11.165151 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:11.165531 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:11.165532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:11.165534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:11.165570 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.165571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.166837 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.166838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.168060 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.168061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.168801 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.168802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.168885 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.168888 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:11.168938 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:04:11.168967 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":114,"thread_start_us":52,"threads_started":1}
I20260925 19:04:11.169363 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.165078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33557 (local address 127.0.0.1:39956)
0925 19:04:11.165619 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:11.165629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:11.165672 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:11.165919 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:11.165920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:11.165922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:11.165957 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.165958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.167079 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.167080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.167953 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.167956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.168841 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.168842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.168899 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.168909 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 19:04:11.168974 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 19:04:11.169012 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":36,"threads_started":1}
I20260925 19:04:11.169603 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.165050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.0.98.1:34289)
0925 19:04:11.165260 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.165264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.165273 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:11.165314 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.165315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.165336 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.165424 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:11.165683 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.166579 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.168053 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.168543 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.168626 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.168683 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.169482 (+ 799us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.169534 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":99,"threads_started":1}
I20260925 19:04:11.170012 18888 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7137f295524386996eecbd9223da29 (DEFAULT_TABLE table=test-table [id=d817c5d1278d4b18b8b3c3a29c14f696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:11.170012 18754 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7137f295524386996eecbd9223da29 (DEFAULT_TABLE table=test-table [id=d817c5d1278d4b18b8b3c3a29c14f696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:11.170167 18620 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7137f295524386996eecbd9223da29 (DEFAULT_TABLE table=test-table [id=d817c5d1278d4b18b8b3c3a29c14f696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:11.170372 18754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7137f295524386996eecbd9223da29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:11.170372 18888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7137f295524386996eecbd9223da29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:11.170466 18620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7137f295524386996eecbd9223da29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:11.172834 18968 tablet_bootstrap.cc:492] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b: Bootstrap starting.
I20260925 19:04:11.173410 18968 tablet_bootstrap.cc:654] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:11.173597 18969 tablet_bootstrap.cc:492] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: Bootstrap starting.
I20260925 19:04:11.174002 18968 log.cc:824] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:11.174288 18969 tablet_bootstrap.cc:654] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:11.174398 18970 tablet_bootstrap.cc:492] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: Bootstrap starting.
I20260925 19:04:11.174786 18968 tablet_bootstrap.cc:492] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b: No bootstrap required, opened a new log
I20260925 19:04:11.174786 18969 log.cc:824] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:11.174860 18968 ts_tablet_manager.cc:1397] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:11.175053 18970 tablet_bootstrap.cc:654] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:11.175506 18969 tablet_bootstrap.cc:492] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: No bootstrap required, opened a new log
I20260925 19:04:11.175586 18969 ts_tablet_manager.cc:1397] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:04:11.175654 18970 log.cc:824] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:11.176832 18970 tablet_bootstrap.cc:492] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: No bootstrap required, opened a new log
I20260925 19:04:11.176903 18970 ts_tablet_manager.cc:1397] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 19:04:11.176977 18968 raft_consensus.cc:348] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 } } peers { permanent_uuid: "5103fd16afe9411ab8d1a2224589e37b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34289 } } peers { permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 } }
I20260925 19:04:11.177152 18968 raft_consensus.cc:374] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:11.177196 18968 raft_consensus.cc:729] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5103fd16afe9411ab8d1a2224589e37b, State: Initialized, Role: FOLLOWER
I20260925 19:04:11.177238 18969 raft_consensus.cc:348] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 } } peers { permanent_uuid: "5103fd16afe9411ab8d1a2224589e37b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34289 } } peers { permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 } }
I20260925 19:04:11.177325 18968 consensus_queue.cc:260] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 } } peers { permanent_uuid: "5103fd16afe9411ab8d1a2224589e37b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34289 } } peers { permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 } }
I20260925 19:04:11.177414 18969 raft_consensus.cc:374] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:11.177459 18969 raft_consensus.cc:729] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02138dac66ef434e87a5c6b420e55e98, State: Initialized, Role: FOLLOWER
I20260925 19:04:11.177595 18968 ts_tablet_manager.cc:1428] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:11.177573 18969 consensus_queue.cc:260] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 } } peers { permanent_uuid: "5103fd16afe9411ab8d1a2224589e37b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34289 } } peers { permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 } }
I20260925 19:04:11.177712 18686 heartbeater.cc:503] Master 127.0.98.62:41349 was elected leader, sending a full tablet report...
I20260925 19:04:11.177845 18969 ts_tablet_manager.cc:1428] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:04:11.177855 18954 heartbeater.cc:503] Master 127.0.98.62:41349 was elected leader, sending a full tablet report...
I20260925 19:04:11.179184 18970 raft_consensus.cc:348] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 } } peers { permanent_uuid: "5103fd16afe9411ab8d1a2224589e37b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34289 } } peers { permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 } }
I20260925 19:04:11.179304 18970 raft_consensus.cc:374] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:11.179327 18970 raft_consensus.cc:729] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27e788b8b4b441a87cb52263c88a889, State: Initialized, Role: FOLLOWER
I20260925 19:04:11.179441 18970 consensus_queue.cc:260] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 } } peers { permanent_uuid: "5103fd16afe9411ab8d1a2224589e37b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34289 } } peers { permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 } }
I20260925 19:04:11.179682 18970 ts_tablet_manager.cc:1428] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:11.179703 18820 heartbeater.cc:503] Master 127.0.98.62:41349 was elected leader, sending a full tablet report...
W20260925 19:04:11.241895 18821 tablet.cc:2369] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:04:11.343238 18687 tablet.cc:2369] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 19:04:11.382275 18955 tablet.cc:2369] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:04:11.405135 18974 raft_consensus.cc:482] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:11.405305 18974 raft_consensus.cc:504] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 } } peers { permanent_uuid: "5103fd16afe9411ab8d1a2224589e37b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34289 } } peers { permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 } }
I20260925 19:04:11.405716 18974 leader_election.cc:288] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b27e788b8b4b441a87cb52263c88a889 (127.0.98.2:43945), 02138dac66ef434e87a5c6b420e55e98 (127.0.98.3:33557)
I20260925 19:04:11.408730 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.405909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43945 (local address 127.0.98.1:43759)
0925 19:04:11.406039 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:11.406050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:11.406112 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:11.406288 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:11.406289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:11.406291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:11.406395 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.406397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.407059 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.407060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.407928 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.407931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.408467 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.408468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.408541 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.408545 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:11.408585 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 19:04:11.408635 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260925 19:04:11.408726 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.405996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43759 (local address 127.0.98.2:43945)
0925 19:04:11.406043 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.406045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.406106 (+ 61us) server_negotiation.cc:407] Connection header received
0925 19:04:11.406179 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.406180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.406194 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.406287 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:11.406467 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.406973 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.408022 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.408380 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.408433 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.408461 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.408613 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.408654 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:04:11.408759 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.405937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33557 (local address 127.0.98.1:36803)
0925 19:04:11.406058 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:11.406068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:11.406120 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:11.406335 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:11.406336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:11.406338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:11.406459 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.406461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.407125 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.407126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.407926 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.407928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.408495 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.408496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.408562 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.408565 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:11.408639 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:04:11.408672 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":8,"thread_start_us":52,"threads_started":1}
I20260925 19:04:11.408763 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.406057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36803 (local address 127.0.98.3:33557)
0925 19:04:11.406110 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.406112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.406123 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:04:11.406195 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.406196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.406204 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.406308 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:11.406550 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.407036 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.408027 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.408388 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.408474 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.408483 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.408656 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.408696 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 19:04:11.409091 18774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7137f295524386996eecbd9223da29" candidate_uuid: "5103fd16afe9411ab8d1a2224589e37b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27e788b8b4b441a87cb52263c88a889" is_pre_election: true
I20260925 19:04:11.409191 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7137f295524386996eecbd9223da29" candidate_uuid: "5103fd16afe9411ab8d1a2224589e37b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02138dac66ef434e87a5c6b420e55e98" is_pre_election: true
I20260925 19:04:11.409271 18774 raft_consensus.cc:2454] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5103fd16afe9411ab8d1a2224589e37b in term 0.
I20260925 19:04:11.409394 18908 raft_consensus.cc:2454] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5103fd16afe9411ab8d1a2224589e37b in term 0.
I20260925 19:04:11.409524 18576 leader_election.cc:302] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5103fd16afe9411ab8d1a2224589e37b, b27e788b8b4b441a87cb52263c88a889; no voters:
I20260925 19:04:11.409675 18974 raft_consensus.cc:2781] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:11.409736 18974 raft_consensus.cc:482] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:11.409758 18974 raft_consensus.cc:3037] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:11.410540 18974 raft_consensus.cc:504] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 } } peers { permanent_uuid: "5103fd16afe9411ab8d1a2224589e37b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34289 } } peers { permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 } }
I20260925 19:04:11.410693 18974 leader_election.cc:288] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [CANDIDATE]: Term 1 election: Requested vote from peers b27e788b8b4b441a87cb52263c88a889 (127.0.98.2:43945), 02138dac66ef434e87a5c6b420e55e98 (127.0.98.3:33557)
I20260925 19:04:11.410853 18774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7137f295524386996eecbd9223da29" candidate_uuid: "5103fd16afe9411ab8d1a2224589e37b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27e788b8b4b441a87cb52263c88a889"
I20260925 19:04:11.410881 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7137f295524386996eecbd9223da29" candidate_uuid: "5103fd16afe9411ab8d1a2224589e37b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02138dac66ef434e87a5c6b420e55e98"
I20260925 19:04:11.410933 18774 raft_consensus.cc:3037] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:11.410993 18908 raft_consensus.cc:3037] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:11.411566 18774 raft_consensus.cc:2454] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5103fd16afe9411ab8d1a2224589e37b in term 1.
I20260925 19:04:11.411751 18908 raft_consensus.cc:2454] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5103fd16afe9411ab8d1a2224589e37b in term 1.
I20260925 19:04:11.411741 18576 leader_election.cc:302] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5103fd16afe9411ab8d1a2224589e37b, b27e788b8b4b441a87cb52263c88a889; no voters:
I20260925 19:04:11.411880 18974 raft_consensus.cc:2781] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:11.412025 18974 raft_consensus.cc:686] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 1 LEADER]: Becoming Leader. State: Replica: 5103fd16afe9411ab8d1a2224589e37b, State: Running, Role: LEADER
I20260925 19:04:11.412128 18974 consensus_queue.cc:237] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 } } peers { permanent_uuid: "5103fd16afe9411ab8d1a2224589e37b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34289 } } peers { permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 } }
I20260925 19:04:11.412914 18495 catalog_manager.cc:5495] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5103fd16afe9411ab8d1a2224589e37b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5103fd16afe9411ab8d1a2224589e37b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5103fd16afe9411ab8d1a2224589e37b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 } health_report { overall_health: UNKNOWN } } }
I20260925 19:04:11.454852 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.452341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.0.98.1:34289)
0925 19:04:11.452409 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.452411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.452438 (+ 27us) server_negotiation.cc:407] Connection header received
0925 19:04:11.452486 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.452486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.452507 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.452587 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:11.452740 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.453214 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.454023 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.454228 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.454367 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.454533 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:11.454618 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:11.454724 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.454753 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.454766 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 19:04:11.456753 18774 raft_consensus.cc:1261] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889 [term 1 FOLLOWER]: Refusing update from remote peer 5103fd16afe9411ab8d1a2224589e37b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:11.456771 18908 raft_consensus.cc:1261] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98 [term 1 FOLLOWER]: Refusing update from remote peer 5103fd16afe9411ab8d1a2224589e37b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:11.457134 18974 consensus_queue.cc:1035] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [LEADER]: Connected to new peer: Peer: permanent_uuid: "02138dac66ef434e87a5c6b420e55e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:11.457264 18979 consensus_queue.cc:1035] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27e788b8b4b441a87cb52263c88a889" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:11.459967 18983 mvcc.cc:205] Tried to move back new op lower bound from 7333327058764103680 to 7333327058584633344. Current Snapshot: MvccSnapshot[applied={T|T < 7333327058764103680}]
I20260925 19:04:11.460397 18984 mvcc.cc:205] Tried to move back new op lower bound from 7333327058764103680 to 7333327058584633344. Current Snapshot: MvccSnapshot[applied={T|T < 7333327058764103680}]
I20260925 19:04:11.461988 18494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:11.462086 18494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:11.462900 18494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4cce2cb918a482eaacf3e5750849abd: Sending DeleteTablet for 3 replicas of tablet 6d7137f295524386996eecbd9223da29
I20260925 19:04:11.463285 18754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d7137f295524386996eecbd9223da29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50872
I20260925 19:04:11.463320 18620 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d7137f295524386996eecbd9223da29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49472
I20260925 19:04:11.463517 18888 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d7137f295524386996eecbd9223da29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39956
I20260925 19:04:11.463577 18992 tablet_replica.cc:330] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b: stopping tablet replica
I20260925 19:04:11.463660 18992 raft_consensus.cc:2229] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:11.463682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 18557
I20260925 19:04:11.463711 18993 tablet_replica.cc:330] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: stopping tablet replica
I20260925 19:04:11.463742 18992 raft_consensus.cc:2258] T 6d7137f295524386996eecbd9223da29 P 5103fd16afe9411ab8d1a2224589e37b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:11.463809 18993 raft_consensus.cc:2229] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:11.463899 18993 raft_consensus.cc:2258] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:11.464044 18991 tablet_replica.cc:330] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: stopping tablet replica
I20260925 19:04:11.464118 18991 raft_consensus.cc:2229] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:11.464247 18991 raft_consensus.cc:2258] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:11.464664 18993 ts_tablet_manager.cc:1905] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:11.464920 18991 ts_tablet_manager.cc:1905] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:11.466480 18991 ts_tablet_manager.cc:1918] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:11.466544 18991 log.cc:1196] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/6d7137f295524386996eecbd9223da29
I20260925 19:04:11.466801 18993 ts_tablet_manager.cc:1918] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:11.466801 18991 ts_tablet_manager.cc:1939] T 6d7137f295524386996eecbd9223da29 P b27e788b8b4b441a87cb52263c88a889: Deleting consensus metadata
I20260925 19:04:11.466871 18993 log.cc:1196] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/6d7137f295524386996eecbd9223da29
I20260925 19:04:11.467124 18993 ts_tablet_manager.cc:1939] T 6d7137f295524386996eecbd9223da29 P 02138dac66ef434e87a5c6b420e55e98: Deleting consensus metadata
I20260925 19:04:11.467231 18482 catalog_manager.cc:4841] TS b27e788b8b4b441a87cb52263c88a889 (127.0.98.2:43945): tablet 6d7137f295524386996eecbd9223da29 (table test-table [id=d817c5d1278d4b18b8b3c3a29c14f696]) successfully deleted
I20260925 19:04:11.467576 18481 catalog_manager.cc:4841] TS 02138dac66ef434e87a5c6b420e55e98 (127.0.98.3:33557): tablet 6d7137f295524386996eecbd9223da29 (table test-table [id=d817c5d1278d4b18b8b3c3a29c14f696]) successfully deleted
W20260925 19:04:11.470335 18481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34289 (error 108)
I20260925 19:04:11.470502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 18691
I20260925 19:04:11.471100 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.470688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34289 (local address 127.0.0.1:49492)
0925 19:04:11.471020 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:11.471061 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
W20260925 19:04:11.471241 18481 catalog_manager.cc:4584] TS 5103fd16afe9411ab8d1a2224589e37b (127.0.98.1:34289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d7137f295524386996eecbd9223da29: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34289: connect: Connection refused (error 111)
I20260925 19:04:11.475184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 18825
I20260925 19:04:11.480484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 18463
2026-09-25T19:04:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2080 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T19:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:11Z Disabled control of system clock
I20260925 19:04:11.507238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43611
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:04:11.585876 18998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:11.586081 18998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:11.586100 18998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:11.587235 18998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:11.587275 18998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:11.587288 18998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:11.587301 18998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:11.587312 18998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:11.588531 18998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:11.588709 18998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:11.588928 18998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:11.593595 19004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:11.593569 19006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:11.593580 19003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:11.593658 18998 server_base.cc:1034] running on GCE node
I20260925 19:04:11.594136 18998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:11.594463 18998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:11.595636 18998 hybrid_clock.cc:648] HybridClock initialized: now 1790363051595624 us; error 28 us; skew 500 ppm
I20260925 19:04:11.596832 18998 webserver.cc:460] Webserver started at http://127.0.98.62:40073/ using document root <none> and password file <none>
I20260925 19:04:11.597082 18998 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:11.597150 18998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:11.597303 18998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:11.598390 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "5f8ec2908ecb459a86cfeb121bf73967"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "698a8ca921c282152a6babbbf0766dd9"
server_key_iv: "54c2a0d065948885100cb5233d3b512b"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.598731 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f8ec2908ecb459a86cfeb121bf73967"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "698a8ca921c282152a6babbbf0766dd9"
server_key_iv: "54c2a0d065948885100cb5233d3b512b"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.600003 18998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:11.600803 19012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.600996 18998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:11.601092 18998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "5f8ec2908ecb459a86cfeb121bf73967"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "698a8ca921c282152a6babbbf0766dd9"
server_key_iv: "54c2a0d065948885100cb5233d3b512b"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.601171 18998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:11.618744 18998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:11.619073 18998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:11.619239 18998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:11.623510 18998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43611
I20260925 19:04:11.623557 19064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43611 every 8 connection(s)
I20260925 19:04:11.623937 18998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:11.624873 19065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:11.626997 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967: Bootstrap starting.
I20260925 19:04:11.627491 19065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:11.627719 19065 log.cc:824] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:11.628314 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967: No bootstrap required, opened a new log
I20260925 19:04:11.629763 19065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ec2908ecb459a86cfeb121bf73967" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } }
I20260925 19:04:11.629899 19065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:11.629921 19065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8ec2908ecb459a86cfeb121bf73967, State: Initialized, Role: FOLLOWER
I20260925 19:04:11.630052 19065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ec2908ecb459a86cfeb121bf73967" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } }
I20260925 19:04:11.630127 19065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:11.630158 19065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:11.630190 19065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:11.630685 19065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ec2908ecb459a86cfeb121bf73967" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } }
I20260925 19:04:11.630764 19065 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8ec2908ecb459a86cfeb121bf73967; no voters:
I20260925 19:04:11.630884 19065 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:11.630954 19068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:11.631127 19065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:11.631146 19068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8ec2908ecb459a86cfeb121bf73967, State: Running, Role: LEADER
I20260925 19:04:11.631201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 18998
I20260925 19:04:11.631237 19068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ec2908ecb459a86cfeb121bf73967" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } }
I20260925 19:04:11.631294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:11.631485 392 external_mini_cluster.cc:1428] Setting key 43a0a6830be8a83f00418191da5c47f3
I20260925 19:04:11.631711 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f8ec2908ecb459a86cfeb121bf73967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ec2908ecb459a86cfeb121bf73967" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } } }
I20260925 19:04:11.631808 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:11.631945 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8ec2908ecb459a86cfeb121bf73967. Latest consensus state: current_term: 1 leader_uuid: "5f8ec2908ecb459a86cfeb121bf73967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8ec2908ecb459a86cfeb121bf73967" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } } }
I20260925 19:04:11.631999 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:11.632428 19077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:11.632951 19077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:11.634526 19077 catalog_manager.cc:1347] Generated new cluster ID: d9792a8ce8224f3faea98949e9e4c0f9
I20260925 19:04:11.634580 19077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:11.638393 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.632120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.0.98.62:43611)
0925 19:04:11.632536 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.632545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.632830 (+ 285us) server_negotiation.cc:407] Connection header received
0925 19:04:11.632981 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.632988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.633068 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.633212 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:11.633445 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.634804 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.635958 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.636584 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.636692 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.637560 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:11.637564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:11.637564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:11.637759 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:11.637798 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:11.637812 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:11.637867 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.638070 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.638145 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":85,"threads_started":1}
I20260925 19:04:11.649621 19077 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:11.650106 19077 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:11.657078 19077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967: Generated new TSK 0
I20260925 19:04:11.657275 19077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:11.659762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43611
--builtin_ntp_servers=127.0.98.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:04:11.743057 19089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:11.743249 19089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:11.743271 19089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:11.744499 19089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:11.744537 19089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:11.744562 19089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:04:11.745865 19089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:11.746119 19089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:11.746460 19089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:11.751080 19097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:11.751180 19094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:11.751281 19095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:11.751525 19089 server_base.cc:1034] running on GCE node
I20260925 19:04:11.751751 19089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:11.752020 19089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:11.753191 19089 hybrid_clock.cc:648] HybridClock initialized: now 1790363051753179 us; error 29 us; skew 500 ppm
I20260925 19:04:11.754520 19089 webserver.cc:460] Webserver started at http://127.0.98.1:46435/ using document root <none> and password file <none>
I20260925 19:04:11.754746 19089 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:11.754810 19089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:11.754976 19089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:11.756045 19089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad7685da041a43de8f239e64db1fd193"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "5014b05e6255dfbdaab951907e1e437b"
server_key_iv: "7383e4a4cedd19aae013e05ec82008fa"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.756419 19089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad7685da041a43de8f239e64db1fd193"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "5014b05e6255dfbdaab951907e1e437b"
server_key_iv: "7383e4a4cedd19aae013e05ec82008fa"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.757797 19089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:11.758634 19103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.758798 19089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.758911 19089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "ad7685da041a43de8f239e64db1fd193"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "5014b05e6255dfbdaab951907e1e437b"
server_key_iv: "7383e4a4cedd19aae013e05ec82008fa"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.759001 19089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:11.779699 19089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:11.780030 19089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:11.780182 19089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:11.780421 19089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:11.780747 19089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:11.780797 19089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.780830 19089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:11.780880 19089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.786761 19089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35059
I20260925 19:04:11.786808 19216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35059 every 8 connection(s)
I20260925 19:04:11.787114 19089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:04:11.792243 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.787790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35069 (local address 127.0.98.62:43611)
0925 19:04:11.787862 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.787863 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.788090 (+ 227us) server_negotiation.cc:407] Connection header received
0925 19:04:11.788599 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.788600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.788616 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.788677 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:11.789323 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.789853 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.790890 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.791095 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.791177 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.791975 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:11.791977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:11.791977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:11.791982 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:11.791994 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:11.791998 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:11.792040 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.792136 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.792165 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:04:11.792477 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.787772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43611 (local address 127.0.98.1:35069)
0925 19:04:11.787973 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:11.788005 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:11.788498 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:11.788747 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:11.788751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:11.788758 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:11.789231 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.789235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.789956 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.789959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.790782 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.790784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.791188 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.791189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.791332 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.791824 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:04:11.791824 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:11.791867 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:04:11.791888 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:11.792051 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:11.792055 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:11.792059 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:11.792137 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 19:04:11.792233 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260925 19:04:11.792980 19217 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43611
I20260925 19:04:11.793053 19217 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:11.793183 19217 heartbeater.cc:511] Master 127.0.98.62:43611 requested a full tablet report, sending...
I20260925 19:04:11.793627 19029 ts_manager.cc:194] Registered new tserver with Master: ad7685da041a43de8f239e64db1fd193 (127.0.98.1:35059)
I20260925 19:04:11.794198 19029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35069
I20260925 19:04:11.794687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 19089
I20260925 19:04:11.794790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:04:11.794976 392 external_mini_cluster.cc:1428] Setting key 7a3e9a74487ff59780937bba54346951
I20260925 19:04:11.795761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43611
--builtin_ntp_servers=127.0.98.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:04:11.874034 19220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:11.874258 19220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:11.874311 19220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:11.875407 19220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:11.875471 19220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:11.875511 19220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:04:11.876835 19220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:11.877082 19220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:11.877362 19220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:11.882323 19225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:11.882337 19228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:11.882323 19226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:11.882354 19220 server_base.cc:1034] running on GCE node
I20260925 19:04:11.882764 19220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:11.883056 19220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:11.884241 19220 hybrid_clock.cc:648] HybridClock initialized: now 1790363051884236 us; error 36 us; skew 500 ppm
I20260925 19:04:11.889909 19220 webserver.cc:460] Webserver started at http://127.0.98.2:39957/ using document root <none> and password file <none>
I20260925 19:04:11.890177 19220 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:11.890249 19220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:11.890431 19220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:11.891512 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "7ae90cbf82c844a28b2988011fa14df6"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "8b8e157323ce7fcdbf9c07967d714d31"
server_key_iv: "1da87f62b112db2f2da64135e4b7a150"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.891877 19220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7ae90cbf82c844a28b2988011fa14df6"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "8b8e157323ce7fcdbf9c07967d714d31"
server_key_iv: "1da87f62b112db2f2da64135e4b7a150"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.893168 19220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:11.893855 19234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.894027 19220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:11.894141 19220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "7ae90cbf82c844a28b2988011fa14df6"
format_stamp: "Formatted at 2026-09-25 19:04:11 on dist-test-slave-49bz"
server_key: "8b8e157323ce7fcdbf9c07967d714d31"
server_key_iv: "1da87f62b112db2f2da64135e4b7a150"
server_key_version: "encryptionkey@0"
I20260925 19:04:11.894213 19220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:11.905881 19220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:11.906167 19220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:11.906340 19220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:11.906565 19220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:11.906901 19220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:11.906955 19220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.906988 19220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:11.907014 19220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:11.913190 19220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43103
I20260925 19:04:11.913273 19347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43103 every 8 connection(s)
I20260925 19:04:11.913676 19220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:04:11.920324 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.914538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51897 (local address 127.0.98.62:43611)
0925 19:04:11.914592 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:11.914594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:11.914887 (+ 293us) server_negotiation.cc:407] Connection header received
0925 19:04:11.915705 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:11.915706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:11.915716 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:11.915816 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:11.916735 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.917470 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.918693 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.918943 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.919019 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.919984 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:11.919985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:11.919985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:11.919991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:11.920008 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:11.920012 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:11.920063 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:11.920185 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 19:04:11.920223 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 19:04:11.920377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 19220
I20260925 19:04:11.920411 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:11.914525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43611 (local address 127.0.98.2:51897)
0925 19:04:11.914747 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:11.914795 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:11.915579 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:11.915903 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:11.915907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:11.915918 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:11.916645 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.916652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.917549 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.917550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.918620 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:11.918623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:11.919039 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:11.919040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:11.919176 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:11.919781 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:04:11.919781 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:11.919825 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:04:11.919847 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:11.920086 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:11.920091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:11.920094 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:11.920171 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 19:04:11.920257 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260925 19:04:11.920465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:04:11.920679 392 external_mini_cluster.cc:1428] Setting key a1a43f5909e455e795b62dbc575b671b
I20260925 19:04:11.921272 19348 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43611
I20260925 19:04:11.921372 19348 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:11.921545 19348 heartbeater.cc:511] Master 127.0.98.62:43611 requested a full tablet report, sending...
I20260925 19:04:11.921579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43611
--builtin_ntp_servers=127.0.98.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:04:11.921941 19029 ts_manager.cc:194] Registered new tserver with Master: 7ae90cbf82c844a28b2988011fa14df6 (127.0.98.2:43103)
I20260925 19:04:11.922482 19029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51897
W20260925 19:04:12.000236 19351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:12.000425 19351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:12.000447 19351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:12.001513 19351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:12.001546 19351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:12.001569 19351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:04:12.002875 19351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:12.003054 19351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:12.003356 19351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:12.008275 19356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:12.008351 19359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:12.008455 19351 server_base.cc:1034] running on GCE node
W20260925 19:04:12.008474 19357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:12.008752 19351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:12.008999 19351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:12.010157 19351 hybrid_clock.cc:648] HybridClock initialized: now 1790363052010136 us; error 36 us; skew 500 ppm
I20260925 19:04:12.011500 19351 webserver.cc:460] Webserver started at http://127.0.98.3:38063/ using document root <none> and password file <none>
I20260925 19:04:12.011759 19351 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:12.011848 19351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:12.011984 19351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:12.013000 19351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "0aa5547c52bb4cccad1a657f1d4f7761"
format_stamp: "Formatted at 2026-09-25 19:04:12 on dist-test-slave-49bz"
server_key: "d6fa24507e0f2b985be6c6c55e61c992"
server_key_iv: "8bc0a843a4bf5656ac09f49db2ad16b7"
server_key_version: "encryptionkey@0"
I20260925 19:04:12.013355 19351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0aa5547c52bb4cccad1a657f1d4f7761"
format_stamp: "Formatted at 2026-09-25 19:04:12 on dist-test-slave-49bz"
server_key: "d6fa24507e0f2b985be6c6c55e61c992"
server_key_iv: "8bc0a843a4bf5656ac09f49db2ad16b7"
server_key_version: "encryptionkey@0"
I20260925 19:04:12.014647 19351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:12.015455 19365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:12.015619 19351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:12.015709 19351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "0aa5547c52bb4cccad1a657f1d4f7761"
format_stamp: "Formatted at 2026-09-25 19:04:12 on dist-test-slave-49bz"
server_key: "d6fa24507e0f2b985be6c6c55e61c992"
server_key_iv: "8bc0a843a4bf5656ac09f49db2ad16b7"
server_key_version: "encryptionkey@0"
I20260925 19:04:12.015786 19351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:12.029409 19351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:12.029675 19351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:12.029829 19351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:12.030046 19351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:12.030391 19351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:12.030441 19351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:12.030483 19351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:12.030510 19351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:12.036347 19351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40149
I20260925 19:04:12.036399 19478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40149 every 8 connection(s)
I20260925 19:04:12.036695 19351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:04:12.041536 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.037366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48389 (local address 127.0.98.62:43611)
0925 19:04:12.037444 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:12.037447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:12.037627 (+ 180us) server_negotiation.cc:407] Connection header received
0925 19:04:12.038067 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:12.038068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:12.038079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:12.038130 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:12.038853 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.039363 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.040301 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.040486 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.040537 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.041336 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:12.041337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:12.041337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:12.041341 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:12.041351 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:12.041354 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:12.041392 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:12.041458 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 19:04:12.041480 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 19:04:12.041702 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.037335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43611 (local address 127.0.98.3:48389)
0925 19:04:12.037510 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:12.037541 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:12.037987 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:12.038180 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:12.038182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:12.038190 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:12.038723 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.038727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.039445 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.039445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.040223 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.040225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.040551 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.040552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.040646 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.041207 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:04:12.041207 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:12.041233 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:04:12.041254 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:12.041394 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:12.041396 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:12.041398 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:12.041443 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 19:04:12.041513 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":98,"threads_started":1}
I20260925 19:04:12.042163 19479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43611
I20260925 19:04:12.042279 19479 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:12.042485 19479 heartbeater.cc:511] Master 127.0.98.62:43611 requested a full tablet report, sending...
I20260925 19:04:12.042863 19029 ts_manager.cc:194] Registered new tserver with Master: 0aa5547c52bb4cccad1a657f1d4f7761 (127.0.98.3:40149)
I20260925 19:04:12.043222 19029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48389
I20260925 19:04:12.046487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 19351
I20260925 19:04:12.046607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:04:12.046818 392 external_mini_cluster.cc:1428] Setting key fcd00e7a542501b271ccecef744be3b8
I20260925 19:04:12.047698 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:04:12.051523 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.048957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.0.98.62:43611)
0925 19:04:12.049016 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:12.049018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:12.049154 (+ 136us) server_negotiation.cc:407] Connection header received
0925 19:04:12.049176 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:12.049177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:12.049186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:12.049231 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:12.049386 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.049805 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.050526 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.050707 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.050752 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.051263 (+ 511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:12.051264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:12.051264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:12.051267 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:12.051278 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:12.051282 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:12.051345 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:12.051409 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 19:04:12.051417 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:04:12.052881 19029 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:12.054360 19029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:12.060078 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.056423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.0.98.3:40149)
0925 19:04:12.056593 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:12.056598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:12.056642 (+ 44us) server_negotiation.cc:407] Connection header received
0925 19:04:12.056756 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:12.056759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:12.056776 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:12.056826 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:12.057059 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.058151 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.059112 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.059667 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.059703 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.059758 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:12.059975 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 19:04:12.060007 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260925 19:04:12.060070 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.056352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40149 (local address 127.0.0.1:50268)
0925 19:04:12.056579 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:12.056588 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:12.056667 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:12.056921 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:12.056922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:12.056924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:12.056969 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.056971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.058227 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.058228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.058978 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.058980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.059805 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.059805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.059863 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.059871 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 19:04:12.059949 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 19:04:12.060007 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":90,"threads_started":1}
I20260925 19:04:12.060335 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.056500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43103 (local address 127.0.0.1:49162)
0925 19:04:12.056648 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:12.056658 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:12.056706 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:12.056989 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:12.056990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:12.056991 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:12.057022 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.057024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.057999 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.057999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.059213 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.059215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.060129 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.060130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.060183 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.060187 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:12.060212 (+ 25us) client_negotiation.cc:240] Negotiation successful
0925 19:04:12.060264 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":58,"threads_started":1}
I20260925 19:04:12.060516 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.056309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35059 (local address 127.0.0.1:37790)
0925 19:04:12.056510 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:12.056546 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:12.056614 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:12.056889 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:12.056892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:12.056894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:12.056957 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.056959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.058199 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.058200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.059261 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.059263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.060131 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.060131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.060371 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.060374 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:12.060406 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 19:04:12.060445 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":193,"thread_start_us":47,"threads_started":1}
I20260925 19:04:12.060756 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.056685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49162 (local address 127.0.98.2:43103)
0925 19:04:12.056844 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:12.056847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:12.056857 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:04:12.056891 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:12.056893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:12.056908 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:12.056981 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:12.057118 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.057920 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.059318 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.060032 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.060499 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.060600 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:12.060625 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 19:04:12.060666 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260925 19:04:12.060911 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.056414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.0.98.1:35059)
0925 19:04:12.056747 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:12.056752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:12.056760 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:12.056796 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:12.056799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:12.056812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:12.056883 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:12.057041 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.058117 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.059337 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.060032 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.060655 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.060751 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:12.060773 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 19:04:12.060812 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":83,"threads_started":1}
I20260925 19:04:12.060981 19413 tablet_service.cc:1467] Processing CreateTablet for tablet 4b99010f8e6a422ea30f9216af8c702e (DEFAULT_TABLE table=test-table [id=a7714d84738b4f3dbbd844a5b5a989cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:12.061349 19413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b99010f8e6a422ea30f9216af8c702e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:12.061328 19282 tablet_service.cc:1467] Processing CreateTablet for tablet 4b99010f8e6a422ea30f9216af8c702e (DEFAULT_TABLE table=test-table [id=a7714d84738b4f3dbbd844a5b5a989cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:12.061491 19151 tablet_service.cc:1467] Processing CreateTablet for tablet 4b99010f8e6a422ea30f9216af8c702e (DEFAULT_TABLE table=test-table [id=a7714d84738b4f3dbbd844a5b5a989cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:12.061699 19282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b99010f8e6a422ea30f9216af8c702e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:12.061774 19151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b99010f8e6a422ea30f9216af8c702e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:12.064141 19493 tablet_bootstrap.cc:492] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: Bootstrap starting.
I20260925 19:04:12.064554 19494 tablet_bootstrap.cc:492] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: Bootstrap starting.
I20260925 19:04:12.064950 19495 tablet_bootstrap.cc:492] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193: Bootstrap starting.
I20260925 19:04:12.065099 19493 tablet_bootstrap.cc:654] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:12.065130 19494 tablet_bootstrap.cc:654] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:12.065435 19494 log.cc:824] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:12.065438 19493 log.cc:824] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:12.065564 19495 tablet_bootstrap.cc:654] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:12.065824 19495 log.cc:824] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:12.066183 19493 tablet_bootstrap.cc:492] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: No bootstrap required, opened a new log
I20260925 19:04:12.066293 19493 ts_tablet_manager.cc:1397] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 19:04:12.066570 19494 tablet_bootstrap.cc:492] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: No bootstrap required, opened a new log
I20260925 19:04:12.066573 19495 tablet_bootstrap.cc:492] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193: No bootstrap required, opened a new log
I20260925 19:04:12.066648 19495 ts_tablet_manager.cc:1397] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:12.066653 19494 ts_tablet_manager.cc:1397] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:12.068651 19494 raft_consensus.cc:348] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } } peers { permanent_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40149 } }
I20260925 19:04:12.068792 19494 raft_consensus.cc:374] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:12.068851 19494 raft_consensus.cc:729] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa5547c52bb4cccad1a657f1d4f7761, State: Initialized, Role: FOLLOWER
I20260925 19:04:12.068961 19494 consensus_queue.cc:260] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } } peers { permanent_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40149 } }
I20260925 19:04:12.068892 19493 raft_consensus.cc:348] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } } peers { permanent_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40149 } }
I20260925 19:04:12.069043 19493 raft_consensus.cc:374] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:12.069083 19493 raft_consensus.cc:729] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ae90cbf82c844a28b2988011fa14df6, State: Initialized, Role: FOLLOWER
I20260925 19:04:12.069018 19495 raft_consensus.cc:348] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } } peers { permanent_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40149 } }
I20260925 19:04:12.069149 19495 raft_consensus.cc:374] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:12.069195 19495 raft_consensus.cc:729] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7685da041a43de8f239e64db1fd193, State: Initialized, Role: FOLLOWER
I20260925 19:04:12.069226 19494 ts_tablet_manager.cc:1428] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:12.069212 19493 consensus_queue.cc:260] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } } peers { permanent_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40149 } }
I20260925 19:04:12.069379 19479 heartbeater.cc:503] Master 127.0.98.62:43611 was elected leader, sending a full tablet report...
I20260925 19:04:12.069340 19495 consensus_queue.cc:260] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } } peers { permanent_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40149 } }
I20260925 19:04:12.069474 19493 ts_tablet_manager.cc:1428] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:12.069589 19495 ts_tablet_manager.cc:1428] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:04:12.069667 19348 heartbeater.cc:503] Master 127.0.98.62:43611 was elected leader, sending a full tablet report...
I20260925 19:04:12.069691 19217 heartbeater.cc:503] Master 127.0.98.62:43611 was elected leader, sending a full tablet report...
I20260925 19:04:12.128566 19499 raft_consensus.cc:482] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:12.128741 19499 raft_consensus.cc:504] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } } peers { permanent_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40149 } }
I20260925 19:04:12.129164 19499 leader_election.cc:288] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad7685da041a43de8f239e64db1fd193 (127.0.98.1:35059), 7ae90cbf82c844a28b2988011fa14df6 (127.0.98.2:43103)
I20260925 19:04:12.132256 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.129312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35059 (local address 127.0.98.3:51845)
0925 19:04:12.129369 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:12.129380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:12.129423 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:12.129653 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:12.129654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:12.129655 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:12.129762 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.129764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.130513 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.130514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.131357 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.131359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.132001 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.132002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.132070 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.132073 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:12.132123 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:04:12.132166 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":3}
I20260925 19:04:12.132294 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.129527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35635 (local address 127.0.98.2:43103)
0925 19:04:12.129585 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:12.129587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:12.129726 (+ 139us) server_negotiation.cc:407] Connection header received
0925 19:04:12.129772 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:12.129773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:12.129784 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:12.129836 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:12.129979 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.130455 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.131484 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.132011 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.132071 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.132108 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:12.132198 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 19:04:12.132237 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 19:04:12.132323 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.129438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43103 (local address 127.0.98.3:35635)
0925 19:04:12.129632 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:12.129644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:12.129720 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:12.129850 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:12.129850 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:12.129852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:12.129913 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.129915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.130549 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.130550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.131338 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:12.131359 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.132079 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.132080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:12.132148 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.132150 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 19:04:12.132195 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 19:04:12.132219 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":110,"threads_started":1}
I20260925 19:04:12.132326 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.129410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51845 (local address 127.0.98.1:35059)
0925 19:04:12.129491 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:12.129493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:12.129503 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:04:12.129535 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:12.129536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:12.129559 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:12.129630 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:12.129858 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.130444 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.131446 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.131935 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.132003 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.132012 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:12.132230 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 19:04:12.132267 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 19:04:12.132711 19302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b99010f8e6a422ea30f9216af8c702e" candidate_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ae90cbf82c844a28b2988011fa14df6" is_pre_election: true
I20260925 19:04:12.132710 19171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b99010f8e6a422ea30f9216af8c702e" candidate_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad7685da041a43de8f239e64db1fd193" is_pre_election: true
I20260925 19:04:12.132893 19302 raft_consensus.cc:2454] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa5547c52bb4cccad1a657f1d4f7761 in term 0.
I20260925 19:04:12.132893 19171 raft_consensus.cc:2454] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa5547c52bb4cccad1a657f1d4f7761 in term 0.
I20260925 19:04:12.133112 19366 leader_election.cc:302] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa5547c52bb4cccad1a657f1d4f7761, ad7685da041a43de8f239e64db1fd193; no voters:
I20260925 19:04:12.133232 19499 raft_consensus.cc:2781] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:12.133282 19499 raft_consensus.cc:482] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:12.133313 19499 raft_consensus.cc:3037] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:12.133985 19499 raft_consensus.cc:504] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } } peers { permanent_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40149 } }
I20260925 19:04:12.134100 19499 leader_election.cc:288] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [CANDIDATE]: Term 1 election: Requested vote from peers ad7685da041a43de8f239e64db1fd193 (127.0.98.1:35059), 7ae90cbf82c844a28b2988011fa14df6 (127.0.98.2:43103)
I20260925 19:04:12.134222 19171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b99010f8e6a422ea30f9216af8c702e" candidate_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad7685da041a43de8f239e64db1fd193"
I20260925 19:04:12.134281 19302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b99010f8e6a422ea30f9216af8c702e" candidate_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ae90cbf82c844a28b2988011fa14df6"
I20260925 19:04:12.134307 19171 raft_consensus.cc:3037] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:12.134361 19302 raft_consensus.cc:3037] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:12.134996 19171 raft_consensus.cc:2454] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa5547c52bb4cccad1a657f1d4f7761 in term 1.
I20260925 19:04:12.135062 19302 raft_consensus.cc:2454] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa5547c52bb4cccad1a657f1d4f7761 in term 1.
I20260925 19:04:12.135174 19366 leader_election.cc:302] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa5547c52bb4cccad1a657f1d4f7761, ad7685da041a43de8f239e64db1fd193; no voters:
I20260925 19:04:12.135260 19499 raft_consensus.cc:2781] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:12.135416 19499 raft_consensus.cc:686] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 1 LEADER]: Becoming Leader. State: Replica: 0aa5547c52bb4cccad1a657f1d4f7761, State: Running, Role: LEADER
I20260925 19:04:12.135505 19499 consensus_queue.cc:237] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } } peers { permanent_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40149 } }
I20260925 19:04:12.136264 19029 catalog_manager.cc:5495] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aa5547c52bb4cccad1a657f1d4f7761 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aa5547c52bb4cccad1a657f1d4f7761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40149 } health_report { overall_health: HEALTHY } } }
I20260925 19:04:12.151607 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.149521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50282 (local address 127.0.98.3:40149)
0925 19:04:12.149580 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:12.149582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:12.149598 (+ 16us) server_negotiation.cc:407] Connection header received
0925 19:04:12.149630 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:12.149631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:12.149647 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:12.149705 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:12.149860 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.150369 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.151016 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:12.151178 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:12.151223 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:12.151346 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:12.151425 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:12.151490 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:12.151532 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 19:04:12.151539 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:04:12.153400 19171 raft_consensus.cc:1261] T 4b99010f8e6a422ea30f9216af8c702e P ad7685da041a43de8f239e64db1fd193 [term 1 FOLLOWER]: Refusing update from remote peer 0aa5547c52bb4cccad1a657f1d4f7761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:12.153400 19302 raft_consensus.cc:1261] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6 [term 1 FOLLOWER]: Refusing update from remote peer 0aa5547c52bb4cccad1a657f1d4f7761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:12.153818 19499 consensus_queue.cc:1035] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad7685da041a43de8f239e64db1fd193" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:12.153976 19503 consensus_queue.cc:1035] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ae90cbf82c844a28b2988011fa14df6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:12.156383 19506 mvcc.cc:205] Tried to move back new op lower bound from 7333327061617405952 to 7333327061547741184. Current Snapshot: MvccSnapshot[applied={T|T < 7333327061617405952}]
I20260925 19:04:12.156513 19508 mvcc.cc:205] Tried to move back new op lower bound from 7333327061617405952 to 7333327061547741184. Current Snapshot: MvccSnapshot[applied={T|T < 7333327061617405952}]
I20260925 19:04:12.157119 19507 mvcc.cc:205] Tried to move back new op lower bound from 7333327061617405952 to 7333327061547741184. Current Snapshot: MvccSnapshot[applied={T|T < 7333327061617405952}]
I20260925 19:04:12.159310 19027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:12.159394 19027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:12.160137 19027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f8ec2908ecb459a86cfeb121bf73967: Sending DeleteTablet for 3 replicas of tablet 4b99010f8e6a422ea30f9216af8c702e
I20260925 19:04:12.160480 19151 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b99010f8e6a422ea30f9216af8c702e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:12 UTC) from {username='slave'} at 127.0.0.1:37790
I20260925 19:04:12.160466 19282 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b99010f8e6a422ea30f9216af8c702e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:12 UTC) from {username='slave'} at 127.0.0.1:49162
I20260925 19:04:12.160667 19413 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b99010f8e6a422ea30f9216af8c702e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:12 UTC) from {username='slave'} at 127.0.0.1:50268
I20260925 19:04:12.160678 19516 tablet_replica.cc:330] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: stopping tablet replica
I20260925 19:04:12.160773 19516 raft_consensus.cc:2229] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:12.160892 19516 raft_consensus.cc:2258] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:12.160909 19518 tablet_replica.cc:330] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: stopping tablet replica
I20260925 19:04:12.161077 19518 raft_consensus.cc:2229] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:12.161125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 19089
I20260925 19:04:12.161273 19518 raft_consensus.cc:2258] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:12.161732 19518 ts_tablet_manager.cc:1905] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:12.162302 19516 ts_tablet_manager.cc:1905] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:12.163923 19518 ts_tablet_manager.cc:1918] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:12.163930 19516 ts_tablet_manager.cc:1918] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:12.163995 19516 log.cc:1196] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/4b99010f8e6a422ea30f9216af8c702e
I20260925 19:04:12.164093 19518 log.cc:1196] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/4b99010f8e6a422ea30f9216af8c702e
I20260925 19:04:12.164255 19516 ts_tablet_manager.cc:1939] T 4b99010f8e6a422ea30f9216af8c702e P 7ae90cbf82c844a28b2988011fa14df6: Deleting consensus metadata
I20260925 19:04:12.164456 19518 ts_tablet_manager.cc:1939] T 4b99010f8e6a422ea30f9216af8c702e P 0aa5547c52bb4cccad1a657f1d4f7761: Deleting consensus metadata
I20260925 19:04:12.164752 19013 catalog_manager.cc:4841] TS 7ae90cbf82c844a28b2988011fa14df6 (127.0.98.2:43103): tablet 4b99010f8e6a422ea30f9216af8c702e (table test-table [id=a7714d84738b4f3dbbd844a5b5a989cc]) successfully deleted
I20260925 19:04:12.164901 19016 catalog_manager.cc:4841] TS 0aa5547c52bb4cccad1a657f1d4f7761 (127.0.98.3:40149): tablet 4b99010f8e6a422ea30f9216af8c702e (table test-table [id=a7714d84738b4f3dbbd844a5b5a989cc]) successfully deleted
W20260925 19:04:12.168354 19013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35059 (error 108)
I20260925 19:04:12.168895 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:12.168748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35059 (local address 127.0.0.1:37798)
0925 19:04:12.168810 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:12.168861 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 19:04:12.168905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 19220
W20260925 19:04:12.169957 19013 catalog_manager.cc:4584] TS ad7685da041a43de8f239e64db1fd193 (127.0.98.1:35059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b99010f8e6a422ea30f9216af8c702e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35059: connect: Connection refused (error 111)
I20260925 19:04:12.173686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 19351
I20260925 19:04:12.178062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 18998
2026-09-25T19:04:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (696 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:12 dist-test-slave-49bz krb5kdc[19522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:12 dist-test-slave-49bz krb5kdc[19522](info): set up 2 sockets
Sep 25 19:04:12 dist-test-slave-49bz krb5kdc[19522](info): commencing operation
krb5kdc: starting...
W20260925 19:04:13.189941 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:13 dist-test-slave-49bz krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:13.301707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:13.380153 19535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:13.380391 19535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:13.380446 19535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:13.381593 19535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:13.381670 19535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:13.381696 19535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:13.381744 19535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:13.381772 19535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:13.383301 19535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:13.383564 19535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:13.383800 19535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:13.388635 19543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:13.388644 19541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:13.388744 19535 server_base.cc:1034] running on GCE node
W20260925 19:04:13.388672 19540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:13.389019 19535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:13.389370 19535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:13.390542 19535 hybrid_clock.cc:648] HybridClock initialized: now 1790363053390520 us; error 36 us; skew 500 ppm
Sep 25 19:04:13 dist-test-slave-49bz krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:13.393208 19535 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:13.393579 19535 webserver.cc:460] Webserver started at http://127.0.98.62:43939/ using document root <none> and password file <none>
I20260925 19:04:13.393764 19535 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:13.393812 19535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:13.393904 19535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:13.394951 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "d8d39fc2fd57448ca3a86b3c2dca1459"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "249510dc80adb06531a46edabcbd7487"
server_key_iv: "4dde1b19922476d17de29fffbba4d888"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.395253 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "d8d39fc2fd57448ca3a86b3c2dca1459"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "249510dc80adb06531a46edabcbd7487"
server_key_iv: "4dde1b19922476d17de29fffbba4d888"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.396577 19535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:13.397289 19550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:13.397464 19535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:13.397544 19535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "d8d39fc2fd57448ca3a86b3c2dca1459"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "249510dc80adb06531a46edabcbd7487"
server_key_iv: "4dde1b19922476d17de29fffbba4d888"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.397598 19535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:13.437534 19535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:13.439996 19535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:13.440109 19535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:13.443816 19535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34875
I20260925 19:04:13.443849 19602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34875 every 8 connection(s)
I20260925 19:04:13.444197 19535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:13.445058 19603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:13.446157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 19535
I20260925 19:04:13.446311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:13.446524 392 external_mini_cluster.cc:1428] Setting key 0ebf3af6aa879a4f1b8e44f096975ead
I20260925 19:04:13.447762 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459: Bootstrap starting.
I20260925 19:04:13.448345 19603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:13.448701 19603 log.cc:824] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:13.449422 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459: No bootstrap required, opened a new log
I20260925 19:04:13.451622 19603 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d39fc2fd57448ca3a86b3c2dca1459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } }
I20260925 19:04:13.451745 19603 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:13.451766 19603 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d39fc2fd57448ca3a86b3c2dca1459, State: Initialized, Role: FOLLOWER
I20260925 19:04:13.451897 19603 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d39fc2fd57448ca3a86b3c2dca1459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } }
I20260925 19:04:13.451967 19603 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:13.451989 19603 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:13.452049 19603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:13.452561 19603 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d39fc2fd57448ca3a86b3c2dca1459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } }
I20260925 19:04:13.452641 19603 leader_election.cc:302] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d39fc2fd57448ca3a86b3c2dca1459; no voters:
I20260925 19:04:13.452838 19603 leader_election.cc:288] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:13.452872 19608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:13.453003 19608 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [term 1 LEADER]: Becoming Leader. State: Replica: d8d39fc2fd57448ca3a86b3c2dca1459, State: Running, Role: LEADER
I20260925 19:04:13.453101 19603 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:13.453081 19608 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d39fc2fd57448ca3a86b3c2dca1459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } }
I20260925 19:04:13.453418 19608 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d39fc2fd57448ca3a86b3c2dca1459. Latest consensus state: current_term: 1 leader_uuid: "d8d39fc2fd57448ca3a86b3c2dca1459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d39fc2fd57448ca3a86b3c2dca1459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } } }
Sep 25 19:04:13 dist-test-slave-49bz krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:13.453497 19608 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:13.453428 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8d39fc2fd57448ca3a86b3c2dca1459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d39fc2fd57448ca3a86b3c2dca1459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } } }
I20260925 19:04:13.453673 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:13.453703 19613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:13.454206 19613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:13.456166 19613 catalog_manager.cc:1347] Generated new cluster ID: c5dc56d73b6a43a28020cdaf9a07c453
I20260925 19:04:13.456216 19613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:13.457636 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.447072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.0.98.62:34875)
0925 19:04:13.447260 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:13.447268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:13.447292 (+ 24us) server_negotiation.cc:407] Connection header received
0925 19:04:13.447696 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:13.447701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:13.447745 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:13.447900 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:13.448546 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.449872 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.450659 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.451346 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.451462 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.454027 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:13.454033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:13.454038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:13.454049 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:13.456105 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:13.456498 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:13.456501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:13.456502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:13.456607 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:13.456812 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:13.456813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:13.456813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:13.457049 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:13.457193 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:13.457304 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 19:04:13.457440 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260925 19:04:13.467293 19613 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:13.467800 19613 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:13.471443 19613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459: Generated new TSK 0
I20260925 19:04:13.471613 19613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 19:04:13.497225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:13.597723 19629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:13.597915 19629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:13.597937 19629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:13.599087 19629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:13.599123 19629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:13.599148 19629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:04:13.600651 19629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:13.600888 19629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:13.601105 19629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:13.605733 19635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:13.605757 19634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:13.605939 19629 server_base.cc:1034] running on GCE node
W20260925 19:04:13.605757 19637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:13.606235 19629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:13.606509 19629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:13.607645 19629 hybrid_clock.cc:648] HybridClock initialized: now 1790363053607633 us; error 28 us; skew 500 ppm
Sep 25 19:04:13 dist-test-slave-49bz krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:13.610105 19629 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 19:04:13.610605 19629 webserver.cc:460] Webserver started at http://127.0.98.1:37825/ using document root <none> and password file <none>
I20260925 19:04:13.610872 19629 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:13.610936 19629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:13.611073 19629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:13.612128 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "6936fe42628d4c2aba46964b9684a729"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "c136d0b545d99095abdac9baff3939f3"
server_key_iv: "531d399b833d18c8c13c7b3596c91cc8"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.612481 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6936fe42628d4c2aba46964b9684a729"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "c136d0b545d99095abdac9baff3939f3"
server_key_iv: "531d399b833d18c8c13c7b3596c91cc8"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.613735 19629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:13.614492 19644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:13.614671 19629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:13.614764 19629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "6936fe42628d4c2aba46964b9684a729"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "c136d0b545d99095abdac9baff3939f3"
server_key_iv: "531d399b833d18c8c13c7b3596c91cc8"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.614840 19629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:13.631155 19629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:13.632899 19629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:13.633072 19629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:13.633278 19629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:13.633567 19629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:13.633618 19629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:13.633652 19629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:13.633679 19629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:13.639375 19629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40055
I20260925 19:04:13.639422 19757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40055 every 8 connection(s)
I20260925 19:04:13.639708 19629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:04:13.642395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 19629
I20260925 19:04:13.642510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:04:13.642696 392 external_mini_cluster.cc:1428] Setting key eb1cfa9f6ff3babf81f0e390d51313d9
Sep 25 19:04:13 dist-test-slave-49bz krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:13.649403 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.640352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40813 (local address 127.0.98.62:34875)
0925 19:04:13.640420 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:13.640422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:13.640645 (+ 223us) server_negotiation.cc:407] Connection header received
0925 19:04:13.641556 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:13.641557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:13.641568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:13.641621 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:13.642221 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.642974 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.644000 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.644225 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.644307 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.646476 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:13.646478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:13.646478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:13.646486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:13.648116 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:13.648474 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:13.648475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:13.648476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:13.648539 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:13.648754 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:13.648755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:13.648755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:13.648901 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:13.648985 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:13.649216 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 19:04:13.649300 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 19:04:13.649446 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.640347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34875 (local address 127.0.98.1:40813)
0925 19:04:13.640528 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:13.640562 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:13.641431 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:13.641697 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:13.641700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:13.641934 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:13.642112 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.642116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.643073 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.643075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.643906 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.643908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.644318 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.644319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.644441 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.644961 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:13.644983 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:13.646392 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:13.648194 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:13.648199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:13.648200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:13.648402 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:13.648605 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:13.648606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:13.648606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:13.648693 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:13.648965 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:13.648968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:13.649076 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 19:04:13.649169 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 19:04:13.649275 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260925 19:04:13.649992 19758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34875
I20260925 19:04:13.650097 19758 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:13.650307 19758 heartbeater.cc:511] Master 127.0.98.62:34875 requested a full tablet report, sending...
I20260925 19:04:13.650893 19567 ts_manager.cc:194] Registered new tserver with Master: 6936fe42628d4c2aba46964b9684a729 (127.0.98.1:40055)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 19:04:13.651724 19567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 19:04:13.668494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:13.745502 19763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:13.745676 19763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:13.745697 19763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:13.746877 19763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:13.746919 19763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:13.746944 19763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:04:13.748344 19763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:13.748562 19763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:13.748759 19763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:13.753443 19768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:13.753429 19769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:13.753624 19763 server_base.cc:1034] running on GCE node
W20260925 19:04:13.753633 19771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:13.753942 19763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:13.754208 19763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:13.755402 19763 hybrid_clock.cc:648] HybridClock initialized: now 1790363053755394 us; error 26 us; skew 500 ppm
Sep 25 19:04:13 dist-test-slave-49bz krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:13.757658 19763 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 19:04:13.758090 19763 webserver.cc:460] Webserver started at http://127.0.98.2:38019/ using document root <none> and password file <none>
I20260925 19:04:13.758364 19763 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:13.758438 19763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:13.758594 19763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:13.759604 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "fa61acef3ef045d5afd8387df97a2275"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "7c16e0f004f08d8eff7afe00a1f5cb39"
server_key_iv: "98ba1df594595dc54d594a5e73bfb040"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.759981 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fa61acef3ef045d5afd8387df97a2275"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "7c16e0f004f08d8eff7afe00a1f5cb39"
server_key_iv: "98ba1df594595dc54d594a5e73bfb040"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.761258 19763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:04:13.761986 19778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:13.762152 19763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:13.762241 19763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "fa61acef3ef045d5afd8387df97a2275"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "7c16e0f004f08d8eff7afe00a1f5cb39"
server_key_iv: "98ba1df594595dc54d594a5e73bfb040"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.762344 19763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:13.791277 19763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:13.793113 19763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:13.793282 19763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:13.793529 19763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:13.793855 19763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:13.793911 19763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:13.793946 19763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:13.793973 19763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:13.799926 19763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36679
I20260925 19:04:13.799968 19891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36679 every 8 connection(s)
I20260925 19:04:13.800271 19763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:04:13.803272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 19763
I20260925 19:04:13.803390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:04:13.803580 392 external_mini_cluster.cc:1428] Setting key 563ccada2edaa7a4d550d42a8bdfe113
Sep 25 19:04:13 dist-test-slave-49bz krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:13.810312 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.800969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41017 (local address 127.0.98.62:34875)
0925 19:04:13.801038 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:13.801041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:13.801283 (+ 242us) server_negotiation.cc:407] Connection header received
0925 19:04:13.802151 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:13.802152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:13.802162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:13.802207 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:13.802855 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.803556 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.804551 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.804872 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.804951 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.807458 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:13.807460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:13.807460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:13.807466 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:13.808814 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:13.809349 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:13.809350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:13.809350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:13.809418 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:13.809684 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:13.809684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:13.809685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:13.809775 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:13.809821 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:13.810140 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 19:04:13.810199 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 19:04:13.810554 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.800990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34875 (local address 127.0.98.2:41017)
0925 19:04:13.801150 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:13.801185 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:13.802067 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:13.802314 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:13.802318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:13.802597 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:13.802776 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.802780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.803656 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.803657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.804471 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.804473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.804960 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.804961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.805053 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.805574 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:13.805585 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:13.807379 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:13.808931 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:13.808938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:13.808940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:13.809272 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:13.809483 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:13.809484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:13.809484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:13.809618 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:13.809835 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:13.809841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:13.810009 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 19:04:13.810123 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 19:04:13.810309 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260925 19:04:13.811040 19892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34875
I20260925 19:04:13.811129 19892 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:13.811311 19892 heartbeater.cc:511] Master 127.0.98.62:34875 requested a full tablet report, sending...
I20260925 19:04:13.811657 19567 ts_manager.cc:194] Registered new tserver with Master: fa61acef3ef045d5afd8387df97a2275 (127.0.98.2:36679)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 19:04:13.812062 19567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 19:04:13.826896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:13.904342 19897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:13.904523 19897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:13.904546 19897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:13.905537 19897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:13.905566 19897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:13.905589 19897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:04:13.906898 19897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:13.907052 19897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:13.907202 19897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:13.911759 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:13.911767 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:13.912031 19897 server_base.cc:1034] running on GCE node
W20260925 19:04:13.911770 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:13.912272 19897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:13.912557 19897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:13.919941 19897 hybrid_clock.cc:648] HybridClock initialized: now 1790363053919918 us; error 38 us; skew 500 ppm
Sep 25 19:04:13 dist-test-slave-49bz krb5kdc[19522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:13.922706 19897 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 19:04:13.923070 19897 webserver.cc:460] Webserver started at http://127.0.98.3:33313/ using document root <none> and password file <none>
I20260925 19:04:13.923251 19897 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:13.923290 19897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:13.923372 19897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:13.924342 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "4fe79cebae7e4ae889b53a69ff3b128e"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "d711335b05d3cb37c02e78cf323fb656"
server_key_iv: "18db45ae537651636604b8f4a44a867f"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.924624 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4fe79cebae7e4ae889b53a69ff3b128e"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "d711335b05d3cb37c02e78cf323fb656"
server_key_iv: "18db45ae537651636604b8f4a44a867f"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.925755 19897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:13.926395 19912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:13.926555 19897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:13.926661 19897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "4fe79cebae7e4ae889b53a69ff3b128e"
format_stamp: "Formatted at 2026-09-25 19:04:13 on dist-test-slave-49bz"
server_key: "d711335b05d3cb37c02e78cf323fb656"
server_key_iv: "18db45ae537651636604b8f4a44a867f"
server_key_version: "encryptionkey@0"
I20260925 19:04:13.926712 19897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:13.943538 19897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:13.945847 19897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:13.945972 19897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:13.946149 19897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:13.946477 19897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:13.946511 19897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:13.946532 19897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:13.946544 19897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:13.951984 19897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45165
I20260925 19:04:13.952060 20025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45165 every 8 connection(s)
I20260925 19:04:13.952364 19897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 19:04:13 dist-test-slave-49bz krb5kdc[19522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:13.960806 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.953124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39653 (local address 127.0.98.62:34875)
0925 19:04:13.953192 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:13.953194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:13.953396 (+ 202us) server_negotiation.cc:407] Connection header received
0925 19:04:13.954471 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:13.954473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:13.954482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:13.954560 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:13.955151 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.955612 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.956510 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.956673 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.956718 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.958599 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:13.958600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:13.958600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:13.958607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:13.959742 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:13.960104 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:13.960105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:13.960105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:13.960152 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:13.960344 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:13.960344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:13.960345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:13.960437 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:13.960484 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:13.960690 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 19:04:13.960745 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:04:13.961009 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.953108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34875 (local address 127.0.98.3:39653)
0925 19:04:13.953284 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:13.953315 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:13.954385 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:13.954627 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:13.954631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:13.954900 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:13.955080 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.955085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.955683 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.955683 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.956444 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.956446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.956735 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.956736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.956825 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.957204 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:13.957210 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:13.958523 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:13.959829 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:13.959834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:13.959836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:13.960041 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:13.960207 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:13.960208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:13.960208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:13.960293 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:13.960487 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:13.960490 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:13.960589 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 19:04:13.960664 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 19:04:13.960767 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260925 19:04:13.961205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 19897
I20260925 19:04:13.961302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:04:13.961494 20026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34875
I20260925 19:04:13.961521 392 external_mini_cluster.cc:1428] Setting key fd3b19712ff9e11dea0452e518159c7c
I20260925 19:04:13.961581 20026 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:13.961766 20026 heartbeater.cc:511] Master 127.0.98.62:34875 requested a full tablet report, sending...
I20260925 19:04:13.962145 19567 ts_manager.cc:194] Registered new tserver with Master: 4fe79cebae7e4ae889b53a69ff3b128e (127.0.98.3:45165)
I20260925 19:04:13.962404 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:04:13.962641 19567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39653
I20260925 19:04:13.970359 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.964069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.0.98.62:34875)
0925 19:04:13.964110 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:13.964112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:13.964132 (+ 20us) server_negotiation.cc:407] Connection header received
0925 19:04:13.964197 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:13.964198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:13.964207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:13.964271 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:13.964999 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.965420 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.966511 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.966663 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.966711 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.968041 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:13.968042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:13.968042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:13.968048 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:13.969422 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:13.969732 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:13.969733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:13.969733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:13.969776 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:13.969926 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:13.969926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:13.969927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:13.970039 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:13.970118 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:13.970202 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 19:04:13.970290 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:04:13.971912 19567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:13.973392 19567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:13.978822 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.975329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45165 (local address 127.0.0.1:58606)
0925 19:04:13.975547 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:13.975582 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:13.975654 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:13.975846 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:13.975848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:13.975850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:13.975892 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.975893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.976980 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.976981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.977754 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.977755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.978566 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.978567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.978640 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.978651 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 19:04:13.978713 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 19:04:13.978760 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":42,"threads_started":1}
I20260925 19:04:13.978833 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.975405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.0.98.3:45165)
0925 19:04:13.975548 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:13.975552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:13.975652 (+ 100us) server_negotiation.cc:407] Connection header received
0925 19:04:13.975746 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:13.975749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:13.975764 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:13.975838 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:13.975944 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.976895 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.977830 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.978482 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.978563 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.978613 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:13.978729 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 19:04:13.978762 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260925 19:04:13.979283 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.975333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40055 (local address 127.0.0.1:50742)
0925 19:04:13.975719 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:13.975728 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:13.975770 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:13.976050 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:13.976050 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:13.976051 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:13.976094 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.976095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.977277 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.977278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.978227 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.978230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.979062 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.979063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.979126 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.979130 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:13.979188 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 19:04:13.979220 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":65,"threads_started":1}
I20260925 19:04:13.979321 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.975555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.0.98.1:40055)
0925 19:04:13.975888 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:13.975892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:13.975901 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:13.975934 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:13.975936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:13.975949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:13.976042 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:13.976206 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.977178 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.978347 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.978989 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.979060 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.979107 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:13.979202 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 19:04:13.979238 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":85,"threads_started":1}
I20260925 19:04:13.979645 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.975349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36679 (local address 127.0.0.1:59246)
0925 19:04:13.976172 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:13.976183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:13.976235 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:13.976528 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:13.976529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:13.976531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:13.976566 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.976567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.977645 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.977646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.978544 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:13.978547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.979417 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.979418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:13.979494 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.979497 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:13.979546 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 19:04:13.979584 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":84,"threads_started":1}
I20260925 19:04:13.979689 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:13.975562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.0.98.2:36679)
0925 19:04:13.975735 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:13.975739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:13.976384 (+ 645us) server_negotiation.cc:407] Connection header received
0925 19:04:13.976438 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:13.976440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:13.976454 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:13.976522 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:13.976619 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.977551 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.978626 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:13.979265 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:13.979324 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:13.979387 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:13.979574 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 19:04:13.979612 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260925 19:04:13.979672 19960 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6a1c9066264bfcb8c506dba28af1a7 (DEFAULT_TABLE table=test-table [id=115f6211d6b64c9a8d1be262c6070109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:13.979992 19960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6a1c9066264bfcb8c506dba28af1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:13.979993 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6a1c9066264bfcb8c506dba28af1a7 (DEFAULT_TABLE table=test-table [id=115f6211d6b64c9a8d1be262c6070109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:13.980247 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6a1c9066264bfcb8c506dba28af1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:13.982913 19826 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6a1c9066264bfcb8c506dba28af1a7 (DEFAULT_TABLE table=test-table [id=115f6211d6b64c9a8d1be262c6070109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:13.983273 19826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6a1c9066264bfcb8c506dba28af1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:13.983491 20043 tablet_bootstrap.cc:492] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729: Bootstrap starting.
I20260925 19:04:13.984057 20043 tablet_bootstrap.cc:654] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:13.984517 20043 log.cc:824] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:13.985178 20043 tablet_bootstrap.cc:492] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729: No bootstrap required, opened a new log
I20260925 19:04:13.985226 20043 ts_tablet_manager.cc:1397] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:13.985780 20046 tablet_bootstrap.cc:492] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: Bootstrap starting.
I20260925 19:04:13.985936 20044 tablet_bootstrap.cc:492] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: Bootstrap starting.
I20260925 19:04:13.986416 20046 tablet_bootstrap.cc:654] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:13.986531 20044 tablet_bootstrap.cc:654] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:13.986931 20046 log.cc:824] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:13.987087 20044 log.cc:824] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:13.987335 20043 raft_consensus.cc:348] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 } } peers { permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 } }
I20260925 19:04:13.987504 20043 raft_consensus.cc:374] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:13.987550 20043 raft_consensus.cc:729] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6936fe42628d4c2aba46964b9684a729, State: Initialized, Role: FOLLOWER
I20260925 19:04:13.987689 20046 tablet_bootstrap.cc:492] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: No bootstrap required, opened a new log
I20260925 19:04:13.987659 20043 consensus_queue.cc:260] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 } } peers { permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 } }
I20260925 19:04:13.987751 20046 ts_tablet_manager.cc:1397] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:04:13.987838 20044 tablet_bootstrap.cc:492] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: No bootstrap required, opened a new log
I20260925 19:04:13.987891 20044 ts_tablet_manager.cc:1397] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:04:13.987910 20043 ts_tablet_manager.cc:1428] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 19:04:13.988022 19758 heartbeater.cc:503] Master 127.0.98.62:34875 was elected leader, sending a full tablet report...
I20260925 19:04:13.989658 20046 raft_consensus.cc:348] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 } } peers { permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 } }
I20260925 19:04:13.989833 20046 raft_consensus.cc:374] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:13.989742 20044 raft_consensus.cc:348] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 } } peers { permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 } }
I20260925 19:04:13.989876 20044 raft_consensus.cc:374] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:13.989869 20046 raft_consensus.cc:729] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa61acef3ef045d5afd8387df97a2275, State: Initialized, Role: FOLLOWER
I20260925 19:04:13.989919 20044 raft_consensus.cc:729] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe79cebae7e4ae889b53a69ff3b128e, State: Initialized, Role: FOLLOWER
I20260925 19:04:13.989992 20046 consensus_queue.cc:260] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 } } peers { permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 } }
I20260925 19:04:13.990015 20044 consensus_queue.cc:260] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 } } peers { permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 } }
I20260925 19:04:13.990285 20046 ts_tablet_manager.cc:1428] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 19:04:13.990286 20044 ts_tablet_manager.cc:1428] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 19:04:13.990311 19892 heartbeater.cc:503] Master 127.0.98.62:34875 was elected leader, sending a full tablet report...
I20260925 19:04:13.990401 20026 heartbeater.cc:503] Master 127.0.98.62:34875 was elected leader, sending a full tablet report...
W20260925 19:04:14.050817 19893 tablet.cc:2369] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:04:14.053498 20051 raft_consensus.cc:482] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:14.053623 20051 raft_consensus.cc:504] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 } } peers { permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 } }
I20260925 19:04:14.054029 20051 leader_election.cc:288] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa61acef3ef045d5afd8387df97a2275 (127.0.98.2:36679), 6936fe42628d4c2aba46964b9684a729 (127.0.98.1:40055)
I20260925 19:04:14.056752 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.054199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36679 (local address 127.0.98.3:34573)
0925 19:04:14.054264 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.054312 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:14.054352 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:14.054518 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:14.054519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:14.054520 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:14.054601 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.054603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.055283 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.055284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.056025 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.056027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.056517 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.056518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.056589 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.056592 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:14.056642 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 19:04:14.056668 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":33}
I20260925 19:04:14.056761 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.054297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34573 (local address 127.0.98.2:36679)
0925 19:04:14.054379 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.054382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.054392 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:04:14.054431 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.054432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.054444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.054510 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:14.054717 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.055212 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.056113 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.056445 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.056493 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.056501 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.056659 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.056702 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 19:04:14.056816 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.054325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40541 (local address 127.0.98.1:40055)
0925 19:04:14.054463 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.054464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.054589 (+ 125us) server_negotiation.cc:407] Connection header received
0925 19:04:14.054636 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.054637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.054651 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.054711 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:14.054864 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.055284 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.056135 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.056468 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.056548 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.056556 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.056716 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.056747 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260925 19:04:14.056871 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.054233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40055 (local address 127.0.98.3:40541)
0925 19:04:14.054513 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.054522 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:14.054574 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:14.054737 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:14.054737 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:14.054739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:14.054797 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.054799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.055354 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.055355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.056033 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.056035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.056568 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.056569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.056641 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.056645 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:14.056697 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 19:04:14.056732 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":155,"threads_started":1}
I20260925 19:04:14.057130 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6a1c9066264bfcb8c506dba28af1a7" candidate_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa61acef3ef045d5afd8387df97a2275" is_pre_election: true
I20260925 19:04:14.057298 19846 raft_consensus.cc:2454] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe79cebae7e4ae889b53a69ff3b128e in term 0.
I20260925 19:04:14.057296 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6a1c9066264bfcb8c506dba28af1a7" candidate_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6936fe42628d4c2aba46964b9684a729" is_pre_election: true
I20260925 19:04:14.057430 19712 raft_consensus.cc:2454] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe79cebae7e4ae889b53a69ff3b128e in term 0.
I20260925 19:04:14.057554 19914 leader_election.cc:302] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe79cebae7e4ae889b53a69ff3b128e, fa61acef3ef045d5afd8387df97a2275; no voters:
I20260925 19:04:14.057677 20051 raft_consensus.cc:2781] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:14.057730 20051 raft_consensus.cc:482] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:14.057752 20051 raft_consensus.cc:3037] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:14.058415 20051 raft_consensus.cc:504] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 } } peers { permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 } }
I20260925 19:04:14.058545 20051 leader_election.cc:288] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [CANDIDATE]: Term 1 election: Requested vote from peers fa61acef3ef045d5afd8387df97a2275 (127.0.98.2:36679), 6936fe42628d4c2aba46964b9684a729 (127.0.98.1:40055)
I20260925 19:04:14.058707 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6a1c9066264bfcb8c506dba28af1a7" candidate_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa61acef3ef045d5afd8387df97a2275"
I20260925 19:04:14.058708 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6a1c9066264bfcb8c506dba28af1a7" candidate_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6936fe42628d4c2aba46964b9684a729"
I20260925 19:04:14.058774 19846 raft_consensus.cc:3037] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:14.058791 19712 raft_consensus.cc:3037] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:14.059401 19846 raft_consensus.cc:2454] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe79cebae7e4ae889b53a69ff3b128e in term 1.
I20260925 19:04:14.059487 19712 raft_consensus.cc:2454] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe79cebae7e4ae889b53a69ff3b128e in term 1.
I20260925 19:04:14.059538 19914 leader_election.cc:302] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe79cebae7e4ae889b53a69ff3b128e, fa61acef3ef045d5afd8387df97a2275; no voters:
I20260925 19:04:14.059612 20051 raft_consensus.cc:2781] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:14.059741 20051 raft_consensus.cc:686] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 1 LEADER]: Becoming Leader. State: Replica: 4fe79cebae7e4ae889b53a69ff3b128e, State: Running, Role: LEADER
I20260925 19:04:14.059831 20051 consensus_queue.cc:237] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 } } peers { permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 } }
I20260925 19:04:14.060590 19567 catalog_manager.cc:5495] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fe79cebae7e4ae889b53a69ff3b128e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe79cebae7e4ae889b53a69ff3b128e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 } health_report { overall_health: UNKNOWN } } }
I20260925 19:04:14.070197 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.067951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.0.98.3:45165)
0925 19:04:14.067992 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.067994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.068002 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:14.068070 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.068070 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.068092 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.068167 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:14.068300 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.068734 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.069473 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.069650 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.069721 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.069881 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:14.069966 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:14.070046 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.070085 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.070104 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:04:14.071707 19846 raft_consensus.cc:1261] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275 [term 1 FOLLOWER]: Refusing update from remote peer 4fe79cebae7e4ae889b53a69ff3b128e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:14.071709 19712 raft_consensus.cc:1261] T 2d6a1c9066264bfcb8c506dba28af1a7 P 6936fe42628d4c2aba46964b9684a729 [term 1 FOLLOWER]: Refusing update from remote peer 4fe79cebae7e4ae889b53a69ff3b128e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:14.071997 20053 consensus_queue.cc:1035] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6936fe42628d4c2aba46964b9684a729" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:14.072124 20051 consensus_queue.cc:1035] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa61acef3ef045d5afd8387df97a2275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:14.074625 20056 mvcc.cc:205] Tried to move back new op lower bound from 7333327069475610624 to 7333327069429743616. Current Snapshot: MvccSnapshot[applied={T|T < 7333327069475610624}]
I20260925 19:04:14.075695 20059 mvcc.cc:205] Tried to move back new op lower bound from 7333327069475610624 to 7333327069429743616. Current Snapshot: MvccSnapshot[applied={T|T < 7333327069475610624}]
I20260925 19:04:14.076727 19566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:14.076821 19566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:14.077507 19566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8d39fc2fd57448ca3a86b3c2dca1459: Sending DeleteTablet for 3 replicas of tablet 2d6a1c9066264bfcb8c506dba28af1a7
I20260925 19:04:14.077850 19960 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6a1c9066264bfcb8c506dba28af1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58606
I20260925 19:04:14.077891 19826 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6a1c9066264bfcb8c506dba28af1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59246
I20260925 19:04:14.078053 20066 tablet_replica.cc:330] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: stopping tablet replica
I20260925 19:04:14.078106 20067 tablet_replica.cc:330] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: stopping tablet replica
I20260925 19:04:14.078115 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6a1c9066264bfcb8c506dba28af1a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50742
I20260925 19:04:14.078188 20067 raft_consensus.cc:2229] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:14.078193 20066 raft_consensus.cc:2229] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:14.078306 20067 raft_consensus.cc:2258] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:14.078316 20066 raft_consensus.cc:2258] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:14.078452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 19629
I20260925 19:04:14.078807 20066 ts_tablet_manager.cc:1905] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:14.079282 20067 ts_tablet_manager.cc:1905] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:14.080924 20067 ts_tablet_manager.cc:1918] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:14.081013 20067 log.cc:1196] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/2d6a1c9066264bfcb8c506dba28af1a7
I20260925 19:04:14.081300 20067 ts_tablet_manager.cc:1939] T 2d6a1c9066264bfcb8c506dba28af1a7 P fa61acef3ef045d5afd8387df97a2275: Deleting consensus metadata
I20260925 19:04:14.081722 19552 catalog_manager.cc:4841] TS fa61acef3ef045d5afd8387df97a2275 (127.0.98.2:36679): tablet 2d6a1c9066264bfcb8c506dba28af1a7 (table test-table [id=115f6211d6b64c9a8d1be262c6070109]) successfully deleted
I20260925 19:04:14.081668 20066 ts_tablet_manager.cc:1918] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:14.081861 20066 log.cc:1196] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/2d6a1c9066264bfcb8c506dba28af1a7
I20260925 19:04:14.082142 20066 ts_tablet_manager.cc:1939] T 2d6a1c9066264bfcb8c506dba28af1a7 P 4fe79cebae7e4ae889b53a69ff3b128e: Deleting consensus metadata
I20260925 19:04:14.082583 19553 catalog_manager.cc:4841] TS 4fe79cebae7e4ae889b53a69ff3b128e (127.0.98.3:45165): tablet 2d6a1c9066264bfcb8c506dba28af1a7 (table test-table [id=115f6211d6b64c9a8d1be262c6070109]) successfully deleted
W20260925 19:04:14.086318 19551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40055 (error 108)
I20260925 19:04:14.086776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 19763
I20260925 19:04:14.086812 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.086672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40055 (local address 127.0.0.1:50750)
0925 19:04:14.086733 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.086777 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 19:04:14.090164 19551 catalog_manager.cc:4584] TS 6936fe42628d4c2aba46964b9684a729 (127.0.98.1:40055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d6a1c9066264bfcb8c506dba28af1a7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40055: connect: Connection refused (error 111)
I20260925 19:04:14.093421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 19897
I20260925 19:04:14.098115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 19535
2026-09-25T19:04:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1920 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T19:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:14Z Disabled control of system clock
I20260925 19:04:14.124732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38925
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:04:14.201297 20073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:14.201514 20073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:14.201534 20073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:14.202579 20073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:14.202611 20073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:14.202623 20073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:14.202636 20073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:14.202647 20073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:14.203927 20073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:14.204109 20073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:14.204337 20073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:04:14.209004 20073 server_base.cc:1034] running on GCE node
W20260925 19:04:14.209004 20078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:14.209008 20079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:14.209102 20081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:14.209563 20073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:14.209863 20073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:14.211038 20073 hybrid_clock.cc:648] HybridClock initialized: now 1790363054211035 us; error 34 us; skew 500 ppm
I20260925 19:04:14.212246 20073 webserver.cc:460] Webserver started at http://127.0.98.62:45767/ using document root <none> and password file <none>
I20260925 19:04:14.212546 20073 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:14.212618 20073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:14.212765 20073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:14.213804 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "3ba4f5a202234d2abb6d06ecb243c2f3"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "92ba569905d9f64b39543943511d7cdd"
server_key_iv: "60083137829a3161f19bbceb9c9816ce"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.214170 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ba4f5a202234d2abb6d06ecb243c2f3"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "92ba569905d9f64b39543943511d7cdd"
server_key_iv: "60083137829a3161f19bbceb9c9816ce"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.215479 20073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:14.216181 20087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:14.216339 20073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:14.216445 20073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "3ba4f5a202234d2abb6d06ecb243c2f3"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "92ba569905d9f64b39543943511d7cdd"
server_key_iv: "60083137829a3161f19bbceb9c9816ce"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.216516 20073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:14.231974 20073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:14.232287 20073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:14.232431 20073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:14.236372 20073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38925
I20260925 19:04:14.236411 20139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38925 every 8 connection(s)
I20260925 19:04:14.236754 20073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:14.237689 20140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:14.238497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 20073
I20260925 19:04:14.238615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:14.238797 392 external_mini_cluster.cc:1428] Setting key b8907cb32ff3dc61137e13697b3756f7
I20260925 19:04:14.240760 20140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3: Bootstrap starting.
I20260925 19:04:14.241418 20140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:14.241716 20140 log.cc:824] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:14.242470 20140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3: No bootstrap required, opened a new log
I20260925 19:04:14.244328 20140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba4f5a202234d2abb6d06ecb243c2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } }
I20260925 19:04:14.244472 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.239314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.0.98.62:38925)
0925 19:04:14.239513 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.239520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.239543 (+ 23us) server_negotiation.cc:407] Connection header received
0925 19:04:14.239969 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.239976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.240032 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.240143 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:14.240378 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.241282 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.242359 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.242799 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.242899 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.243600 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:14.243603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:14.243604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:14.243764 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:14.243794 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:14.243805 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:14.243870 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.243964 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.244022 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260925 19:04:14.244477 20140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:14.244591 20140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba4f5a202234d2abb6d06ecb243c2f3, State: Initialized, Role: FOLLOWER
I20260925 19:04:14.244745 20140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba4f5a202234d2abb6d06ecb243c2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } }
I20260925 19:04:14.244832 20140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:14.244863 20140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:14.244906 20140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:14.245410 20140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba4f5a202234d2abb6d06ecb243c2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } }
I20260925 19:04:14.245493 20140 leader_election.cc:302] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ba4f5a202234d2abb6d06ecb243c2f3; no voters:
I20260925 19:04:14.245609 20140 leader_election.cc:288] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:14.245676 20145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:14.245834 20145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba4f5a202234d2abb6d06ecb243c2f3, State: Running, Role: LEADER
I20260925 19:04:14.245893 20140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:14.245922 20145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba4f5a202234d2abb6d06ecb243c2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } }
I20260925 19:04:14.246307 20146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ba4f5a202234d2abb6d06ecb243c2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba4f5a202234d2abb6d06ecb243c2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } } }
I20260925 19:04:14.246310 20147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba4f5a202234d2abb6d06ecb243c2f3. Latest consensus state: current_term: 1 leader_uuid: "3ba4f5a202234d2abb6d06ecb243c2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba4f5a202234d2abb6d06ecb243c2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } } }
I20260925 19:04:14.246419 20146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:14.246426 20147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:14.246634 20151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:14.247153 20151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:14.248593 20151 catalog_manager.cc:1347] Generated new cluster ID: 8a439b10007f4662b9c58a33755f57bf
I20260925 19:04:14.248653 20151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:14.257900 20151 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:14.258481 20151 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:14.263136 20151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3: Generated new TSK 0
I20260925 19:04:14.263324 20151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:14.267109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38925
--builtin_ntp_servers=127.0.98.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 19:04:14.346096 20164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:14.346342 20164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:14.346380 20164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:14.347723 20164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:14.347779 20164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:14.347820 20164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 19:04:14.349123 20164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:14.349409 20164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:14.349713 20164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:14.354799 20169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:14.354809 20172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:14.354899 20170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:14.355213 20164 server_base.cc:1034] running on GCE node
I20260925 19:04:14.355407 20164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:14.355698 20164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:14.356864 20164 hybrid_clock.cc:648] HybridClock initialized: now 1790363054356852 us; error 26 us; skew 500 ppm
I20260925 19:04:14.358314 20164 webserver.cc:460] Webserver started at http://127.0.98.1:40679/ using document root <none> and password file <none>
I20260925 19:04:14.358594 20164 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:14.358665 20164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:14.358821 20164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:14.360150 20164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/data/instance:
uuid: "2e5c619a999d443db3065677b83044cc"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "f0e7a1a896072318b764e936025093cf"
server_key_iv: "c6467c3bd4719ed4a00c3235c620767c"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.360523 20164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2e5c619a999d443db3065677b83044cc"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "f0e7a1a896072318b764e936025093cf"
server_key_iv: "c6467c3bd4719ed4a00c3235c620767c"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.362102 20164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 19:04:14.363077 20178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:14.363281 20164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:14.363368 20164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/wal
uuid: "2e5c619a999d443db3065677b83044cc"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "f0e7a1a896072318b764e936025093cf"
server_key_iv: "c6467c3bd4719ed4a00c3235c620767c"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.363440 20164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:14.400699 20164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:14.400974 20164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:14.401072 20164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:14.401259 20164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:14.401613 20164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:14.401646 20164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:14.401700 20164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:14.401726 20164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:14.407279 20164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45585
I20260925 19:04:14.407323 20291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45585 every 8 connection(s)
I20260925 19:04:14.407629 20164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/data/info.pb
I20260925 19:04:14.412194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 20164
I20260925 19:04:14.412325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-0/wal/instance
I20260925 19:04:14.412585 392 external_mini_cluster.cc:1428] Setting key dacd8b82bc2d09329d4ec31c287ab9e5
I20260925 19:04:14.412621 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.408310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41179 (local address 127.0.98.62:38925)
0925 19:04:14.408394 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.408396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.408595 (+ 199us) server_negotiation.cc:407] Connection header received
0925 19:04:14.409046 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.409048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.409062 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.409116 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:14.409757 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.410312 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.411258 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.411450 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.411511 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.412302 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:14.412304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:14.412304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:14.412310 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:14.412327 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:14.412332 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:14.412383 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.412504 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.412534 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 19:04:14.412822 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.408267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38925 (local address 127.0.98.1:41179)
0925 19:04:14.408474 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.408507 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:14.408964 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:14.409191 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:14.409194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:14.409202 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:14.409681 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.409684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.410393 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.410394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.411155 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.411157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.411526 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.411527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.411631 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.412156 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:04:14.412157 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:14.412183 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:04:14.412204 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:14.412402 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:14.412407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:14.412410 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:14.412482 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:04:14.412581 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260925 19:04:14.413461 20292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38925
I20260925 19:04:14.413550 20292 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:14.413549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38925
--builtin_ntp_servers=127.0.98.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:04:14.413740 20292 heartbeater.cc:511] Master 127.0.98.62:38925 requested a full tablet report, sending...
I20260925 19:04:14.414228 20104 ts_manager.cc:194] Registered new tserver with Master: 2e5c619a999d443db3065677b83044cc (127.0.98.1:45585)
I20260925 19:04:14.414855 20104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41179
W20260925 19:04:14.494606 20295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:14.494796 20295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:14.494817 20295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:14.495962 20295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:14.496001 20295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:14.496052 20295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 19:04:14.497366 20295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:14.497603 20295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:14.497918 20295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:14.502770 20300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:14.502789 20303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:14.502782 20301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:14.503072 20295 server_base.cc:1034] running on GCE node
I20260925 19:04:14.503337 20295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:14.503575 20295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:14.504712 20295 hybrid_clock.cc:648] HybridClock initialized: now 1790363054504700 us; error 33 us; skew 500 ppm
I20260925 19:04:14.505946 20295 webserver.cc:460] Webserver started at http://127.0.98.2:37585/ using document root <none> and password file <none>
I20260925 19:04:14.506194 20295 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:14.506285 20295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:14.506428 20295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:14.507488 20295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/data/instance:
uuid: "f6202d3ca06341ba872c37be5c58f5ff"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "5c2dadf8f80520924c6ded8cb4bdadac"
server_key_iv: "7d68a5cc2647c8c9fa68c5909cf47306"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.507869 20295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f6202d3ca06341ba872c37be5c58f5ff"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "5c2dadf8f80520924c6ded8cb4bdadac"
server_key_iv: "7d68a5cc2647c8c9fa68c5909cf47306"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.509143 20295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:14.509819 20309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:14.509984 20295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:14.510079 20295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/wal
uuid: "f6202d3ca06341ba872c37be5c58f5ff"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "5c2dadf8f80520924c6ded8cb4bdadac"
server_key_iv: "7d68a5cc2647c8c9fa68c5909cf47306"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.510160 20295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:14.538645 20295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:14.538980 20295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:14.539134 20295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:14.539373 20295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:14.539695 20295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:14.539746 20295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:14.539785 20295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:14.539811 20295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:14.545825 20295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36659
I20260925 19:04:14.545848 20422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36659 every 8 connection(s)
I20260925 19:04:14.546341 20295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/data/info.pb
I20260925 19:04:14.548586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 20295
I20260925 19:04:14.548702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/wal/instance
I20260925 19:04:14.548894 392 external_mini_cluster.cc:1428] Setting key 760787d2d22f0ab86647c7a69e978786
I20260925 19:04:14.549815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38925
--builtin_ntp_servers=127.0.98.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 19:04:14.551779 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.547081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39265 (local address 127.0.98.62:38925)
0925 19:04:14.547155 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.547158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.547372 (+ 214us) server_negotiation.cc:407] Connection header received
0925 19:04:14.547856 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.547857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.547866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.547921 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:14.548571 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.549238 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.550354 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.550557 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.550636 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.551537 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:14.551538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:14.551538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:14.551542 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:14.551557 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:14.551560 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:14.551605 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.551681 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.551705 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 19:04:14.552002 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.547040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38925 (local address 127.0.98.2:39265)
0925 19:04:14.547254 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.547296 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:14.547776 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:14.547984 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:14.547987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:14.547995 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:14.548480 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.548485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.549334 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.549336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.550210 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.550213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.550646 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.550647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.550772 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.551395 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:04:14.551396 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:14.551424 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:04:14.551444 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:14.551604 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:14.551609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:14.551612 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:14.551672 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 19:04:14.551768 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
I20260925 19:04:14.552439 20423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38925
I20260925 19:04:14.552549 20423 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:14.552728 20423 heartbeater.cc:511] Master 127.0.98.62:38925 requested a full tablet report, sending...
I20260925 19:04:14.553098 20104 ts_manager.cc:194] Registered new tserver with Master: f6202d3ca06341ba872c37be5c58f5ff (127.0.98.2:36659)
I20260925 19:04:14.553424 20104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39265
W20260925 19:04:14.630013 20426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:14.630223 20426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:14.630254 20426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:14.631314 20426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:14.631373 20426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:14.631412 20426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 19:04:14.633106 20426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:14.633368 20426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:14.633646 20426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:14.638689 20432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:14.638785 20426 server_base.cc:1034] running on GCE node
W20260925 19:04:14.638686 20431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:14.638686 20434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:14.639127 20426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:14.639400 20426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:14.640571 20426 hybrid_clock.cc:648] HybridClock initialized: now 1790363054640550 us; error 35 us; skew 500 ppm
I20260925 19:04:14.641747 20426 webserver.cc:460] Webserver started at http://127.0.98.3:45975/ using document root <none> and password file <none>
I20260925 19:04:14.641981 20426 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:14.642052 20426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:14.642201 20426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:14.643281 20426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/data/instance:
uuid: "667946fc07024c1f8550e4251605ad43"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "133621499cda5c26ddbabc629f9caa6f"
server_key_iv: "198a18f37338b5654a8ab8556b2e1f48"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.643642 20426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/wal/instance:
uuid: "667946fc07024c1f8550e4251605ad43"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "133621499cda5c26ddbabc629f9caa6f"
server_key_iv: "198a18f37338b5654a8ab8556b2e1f48"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.645016 20426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:14.645717 20440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:14.645895 20426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:14.646020 20426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/wal
uuid: "667946fc07024c1f8550e4251605ad43"
format_stamp: "Formatted at 2026-09-25 19:04:14 on dist-test-slave-49bz"
server_key: "133621499cda5c26ddbabc629f9caa6f"
server_key_iv: "198a18f37338b5654a8ab8556b2e1f48"
server_key_version: "encryptionkey@0"
I20260925 19:04:14.646102 20426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:14.666684 20426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:14.667057 20426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:14.667214 20426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:14.667472 20426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 19:04:14.667812 20426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 19:04:14.667868 20426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:14.667908 20426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 19:04:14.667934 20426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:14.673414 20426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39627
I20260925 19:04:14.673465 20553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39627 every 8 connection(s)
I20260925 19:04:14.673780 20426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/data/info.pb
I20260925 19:04:14.674814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 20426
I20260925 19:04:14.674933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/wal/instance
I20260925 19:04:14.675112 392 external_mini_cluster.cc:1428] Setting key 391c0b63b6f0760cf7909648b5b68045
I20260925 19:04:14.678793 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.674542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33777 (local address 127.0.98.62:38925)
0925 19:04:14.674610 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.674612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.674823 (+ 211us) server_negotiation.cc:407] Connection header received
0925 19:04:14.675413 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.675415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.675426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.675492 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:14.676322 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.676783 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.677674 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.677842 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.677889 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.678611 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:14.678612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:14.678612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:14.678616 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:14.678628 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:14.678632 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:14.678669 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.678720 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.678739 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 19:04:14.678993 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.674529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38925 (local address 127.0.98.3:33777)
0925 19:04:14.674702 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.674733 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:14.675306 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:14.675586 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:14.675592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:14.675601 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:14.676253 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.676259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.676861 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.676861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.677604 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.677606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.677913 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.677914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.678024 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.678493 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 19:04:14.678493 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:14.678520 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 19:04:14.678539 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:14.678675 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:14.678677 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:14.678680 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:14.678721 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 19:04:14.678791 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260925 19:04:14.679473 20554 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38925
I20260925 19:04:14.679586 20554 heartbeater.cc:464] Registering TS with master...
I20260925 19:04:14.679755 20554 heartbeater.cc:511] Master 127.0.98.62:38925 requested a full tablet report, sending...
I20260925 19:04:14.680140 20104 ts_manager.cc:194] Registered new tserver with Master: 667946fc07024c1f8550e4251605ad43 (127.0.98.3:39627)
I20260925 19:04:14.680526 20104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33777
I20260925 19:04:14.686966 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 19:04:14.690996 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.688315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.0.98.62:38925)
0925 19:04:14.688348 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.688350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.688507 (+ 157us) server_negotiation.cc:407] Connection header received
0925 19:04:14.688532 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.688533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.688541 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.688592 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:14.688722 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.689181 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.689889 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.690091 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.690142 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.690798 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:14.690800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:14.690800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:14.690804 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:14.690817 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:14.690822 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:14.690866 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.690924 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.690943 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:04:14.692448 20104 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 19:04:14.694018 20104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 19:04:14.700525 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.696357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.0.98.3:39627)
0925 19:04:14.696561 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.696566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.696705 (+ 139us) server_negotiation.cc:407] Connection header received
0925 19:04:14.696779 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.696782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.696797 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.696874 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:14.697145 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.698255 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.699427 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.700123 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.700198 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.700283 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.700408 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.700451 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260925 19:04:14.700536 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.696189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39627 (local address 127.0.0.1:58430)
0925 19:04:14.696630 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.696640 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:14.696712 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:14.696926 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:14.696929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:14.696931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:14.697060 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.697061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.698365 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.698366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.699336 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.699339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.700215 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.700216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.700298 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.700314 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 19:04:14.700388 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:04:14.700447 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":75,"thread_start_us":78,"threads_started":1}
I20260925 19:04:14.700821 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.696528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.0.98.1:45585)
0925 19:04:14.696755 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.696758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.696767 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:14.696809 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.696810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.696822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.696951 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:14.697098 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.698240 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.699453 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.700125 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.700198 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.700284 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.700680 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.700742 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260925 19:04:14.701362 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.696390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36659 (local address 127.0.0.1:58112)
0925 19:04:14.696795 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.696803 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:14.696850 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:14.697192 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:14.697193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:14.697195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:14.697234 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.697235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.698112 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.698113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.699383 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.699386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.701125 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.701126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.701196 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.701200 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 19:04:14.701247 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 19:04:14.701286 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260925 19:04:14.701397 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.696516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.0.98.2:36659)
0925 19:04:14.696702 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.696707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.696868 (+ 161us) server_negotiation.cc:407] Connection header received
0925 19:04:14.697036 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.697039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.697054 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.697124 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:14.697321 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.698007 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.700319 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.701044 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.701138 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.701235 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.701263 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.701303 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260925 19:04:14.701320 20488 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc9c429d69343ab8d59435d09454161 (DEFAULT_TABLE table=test-table [id=e3fa8a692e4048d3a0bdac317db271b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:14.701713 20488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc9c429d69343ab8d59435d09454161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:14.702033 20357 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc9c429d69343ab8d59435d09454161 (DEFAULT_TABLE table=test-table [id=e3fa8a692e4048d3a0bdac317db271b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:14.702371 20357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc9c429d69343ab8d59435d09454161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:14.700536 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.696176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45585 (local address 127.0.0.1:59770)
0925 19:04:14.696286 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.696324 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:14.696406 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:14.696948 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:14.696948 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:14.696949 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:14.697011 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.697012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.698337 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.698338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.699385 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.699386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.700214 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.700215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.700298 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.700314 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 19:04:14.700388 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 19:04:14.700450 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":23,"thread_start_us":69,"threads_started":1}
I20260925 19:04:14.704524 20226 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc9c429d69343ab8d59435d09454161 (DEFAULT_TABLE table=test-table [id=e3fa8a692e4048d3a0bdac317db271b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 19:04:14.704809 20226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dc9c429d69343ab8d59435d09454161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:14.705163 20568 tablet_bootstrap.cc:492] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: Bootstrap starting.
I20260925 19:04:14.705570 20569 tablet_bootstrap.cc:492] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: Bootstrap starting.
I20260925 19:04:14.706003 20568 tablet_bootstrap.cc:654] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:14.706142 20569 tablet_bootstrap.cc:654] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:14.706382 20568 log.cc:824] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:14.706490 20569 log.cc:824] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:14.707156 20568 tablet_bootstrap.cc:492] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: No bootstrap required, opened a new log
I20260925 19:04:14.707218 20569 tablet_bootstrap.cc:492] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: No bootstrap required, opened a new log
I20260925 19:04:14.707229 20568 ts_tablet_manager.cc:1397] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:04:14.707266 20569 ts_tablet_manager.cc:1397] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 19:04:14.707994 20572 tablet_bootstrap.cc:492] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc: Bootstrap starting.
I20260925 19:04:14.708626 20572 tablet_bootstrap.cc:654] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:14.708909 20572 log.cc:824] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:14.709591 20572 tablet_bootstrap.cc:492] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc: No bootstrap required, opened a new log
I20260925 19:04:14.709673 20572 ts_tablet_manager.cc:1397] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 19:04:14.709748 20568 raft_consensus.cc:348] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 } } peers { permanent_uuid: "667946fc07024c1f8550e4251605ad43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 } }
I20260925 19:04:14.709748 20569 raft_consensus.cc:348] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 } } peers { permanent_uuid: "667946fc07024c1f8550e4251605ad43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 } }
I20260925 19:04:14.709919 20568 raft_consensus.cc:374] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:14.709919 20569 raft_consensus.cc:374] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:14.709956 20569 raft_consensus.cc:729] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6202d3ca06341ba872c37be5c58f5ff, State: Initialized, Role: FOLLOWER
I20260925 19:04:14.709956 20568 raft_consensus.cc:729] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667946fc07024c1f8550e4251605ad43, State: Initialized, Role: FOLLOWER
I20260925 19:04:14.710119 20568 consensus_queue.cc:260] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 } } peers { permanent_uuid: "667946fc07024c1f8550e4251605ad43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 } }
I20260925 19:04:14.710098 20569 consensus_queue.cc:260] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 } } peers { permanent_uuid: "667946fc07024c1f8550e4251605ad43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 } }
I20260925 19:04:14.710384 20569 ts_tablet_manager.cc:1428] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:04:14.710384 20568 ts_tablet_manager.cc:1428] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:04:14.710521 20423 heartbeater.cc:503] Master 127.0.98.62:38925 was elected leader, sending a full tablet report...
I20260925 19:04:14.710815 20554 heartbeater.cc:503] Master 127.0.98.62:38925 was elected leader, sending a full tablet report...
I20260925 19:04:14.712201 20572 raft_consensus.cc:348] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 } } peers { permanent_uuid: "667946fc07024c1f8550e4251605ad43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 } }
I20260925 19:04:14.712375 20572 raft_consensus.cc:374] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:14.712430 20572 raft_consensus.cc:729] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5c619a999d443db3065677b83044cc, State: Initialized, Role: FOLLOWER
I20260925 19:04:14.712546 20572 consensus_queue.cc:260] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 } } peers { permanent_uuid: "667946fc07024c1f8550e4251605ad43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 } }
I20260925 19:04:14.712786 20572 ts_tablet_manager.cc:1428] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 19:04:14.712831 20292 heartbeater.cc:503] Master 127.0.98.62:38925 was elected leader, sending a full tablet report...
W20260925 19:04:14.797029 20424 tablet.cc:2369] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 19:04:14.833695 20574 raft_consensus.cc:482] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 19:04:14.833858 20574 raft_consensus.cc:504] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 } } peers { permanent_uuid: "667946fc07024c1f8550e4251605ad43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 } }
I20260925 19:04:14.834290 20574 leader_election.cc:288] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e5c619a999d443db3065677b83044cc (127.0.98.1:45585), f6202d3ca06341ba872c37be5c58f5ff (127.0.98.2:36659)
I20260925 19:04:14.837219 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.834580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32869 (local address 127.0.98.1:45585)
0925 19:04:14.834629 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.834632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.834665 (+ 33us) server_negotiation.cc:407] Connection header received
0925 19:04:14.834695 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.834696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.834707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.834755 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:14.834957 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.835526 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.836482 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.836824 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.836912 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.836922 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.837109 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.837146 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:04:14.837265 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.834610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53163 (local address 127.0.98.2:36659)
0925 19:04:14.834664 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.834666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.834822 (+ 156us) server_negotiation.cc:407] Connection header received
0925 19:04:14.834856 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.834857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.834868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.834961 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 19:04:14.835125 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.835592 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.836528 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.836865 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.836963 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.836973 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.837161 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.837198 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:04:14.837282 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.834563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36659 (local address 127.0.98.3:53163)
0925 19:04:14.834709 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.834723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:14.834786 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:14.834996 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:14.834996 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:14.834997 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:14.835044 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.835045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.835694 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.835695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.836409 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.836411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.836971 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.836972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.837070 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.837073 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 19:04:14.837153 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 19:04:14.837188 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260925 19:04:14.837473 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.834507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45585 (local address 127.0.98.3:32869)
0925 19:04:14.834568 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.834579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:14.834624 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:14.834782 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:14.834783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:14.834785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 19:04:14.834857 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.834859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.835635 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.835636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.836382 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:14.836384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.836913 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.836914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:14.836990 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.837011 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 19:04:14.837103 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 19:04:14.837146 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":9}
I20260925 19:04:14.837680 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc9c429d69343ab8d59435d09454161" candidate_uuid: "667946fc07024c1f8550e4251605ad43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6202d3ca06341ba872c37be5c58f5ff" is_pre_election: true
I20260925 19:04:14.837826 20377 raft_consensus.cc:2454] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667946fc07024c1f8550e4251605ad43 in term 0.
I20260925 19:04:14.837839 20246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc9c429d69343ab8d59435d09454161" candidate_uuid: "667946fc07024c1f8550e4251605ad43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5c619a999d443db3065677b83044cc" is_pre_election: true
I20260925 19:04:14.837999 20246 raft_consensus.cc:2454] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667946fc07024c1f8550e4251605ad43 in term 0.
I20260925 19:04:14.838078 20443 leader_election.cc:302] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667946fc07024c1f8550e4251605ad43, f6202d3ca06341ba872c37be5c58f5ff; no voters:
I20260925 19:04:14.838204 20574 raft_consensus.cc:2781] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 19:04:14.838276 20574 raft_consensus.cc:482] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 19:04:14.838306 20574 raft_consensus.cc:3037] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:14.839041 20574 raft_consensus.cc:504] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 } } peers { permanent_uuid: "667946fc07024c1f8550e4251605ad43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 } }
I20260925 19:04:14.839246 20574 leader_election.cc:288] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [CANDIDATE]: Term 1 election: Requested vote from peers 2e5c619a999d443db3065677b83044cc (127.0.98.1:45585), f6202d3ca06341ba872c37be5c58f5ff (127.0.98.2:36659)
I20260925 19:04:14.839339 20377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc9c429d69343ab8d59435d09454161" candidate_uuid: "667946fc07024c1f8550e4251605ad43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6202d3ca06341ba872c37be5c58f5ff"
I20260925 19:04:14.839417 20246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc9c429d69343ab8d59435d09454161" candidate_uuid: "667946fc07024c1f8550e4251605ad43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5c619a999d443db3065677b83044cc"
I20260925 19:04:14.839440 20377 raft_consensus.cc:3037] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:14.839478 20246 raft_consensus.cc:3037] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:14.840315 20246 raft_consensus.cc:2454] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667946fc07024c1f8550e4251605ad43 in term 1.
I20260925 19:04:14.840339 20377 raft_consensus.cc:2454] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667946fc07024c1f8550e4251605ad43 in term 1.
I20260925 19:04:14.840485 20443 leader_election.cc:302] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667946fc07024c1f8550e4251605ad43, f6202d3ca06341ba872c37be5c58f5ff; no voters:
I20260925 19:04:14.840646 20574 raft_consensus.cc:2781] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:14.840792 20574 raft_consensus.cc:686] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 1 LEADER]: Becoming Leader. State: Replica: 667946fc07024c1f8550e4251605ad43, State: Running, Role: LEADER
I20260925 19:04:14.840885 20574 consensus_queue.cc:237] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 } } peers { permanent_uuid: "667946fc07024c1f8550e4251605ad43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 } }
I20260925 19:04:14.841715 20102 catalog_manager.cc:5495] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 667946fc07024c1f8550e4251605ad43 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "667946fc07024c1f8550e4251605ad43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "667946fc07024c1f8550e4251605ad43" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 } health_report { overall_health: UNKNOWN } } }
I20260925 19:04:14.881024 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.878882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.0.98.3:39627)
0925 19:04:14.878941 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:14.878943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:14.878950 (+ 7us) server_negotiation.cc:407] Connection header received
0925 19:04:14.879023 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:14.879023 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:14.879041 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:14.879100 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:14.879217 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.879668 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.880364 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:14.880534 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:14.880580 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:14.880708 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:14.880801 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:14.880860 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:14.880896 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 19:04:14.880911 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:04:14.882989 20377 raft_consensus.cc:1261] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff [term 1 FOLLOWER]: Refusing update from remote peer 667946fc07024c1f8550e4251605ad43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:14.882989 20246 raft_consensus.cc:1261] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc [term 1 FOLLOWER]: Refusing update from remote peer 667946fc07024c1f8550e4251605ad43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:14.883375 20574 consensus_queue.cc:1035] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6202d3ca06341ba872c37be5c58f5ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:14.883526 20578 consensus_queue.cc:1035] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5c619a999d443db3065677b83044cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:14.885885 20581 mvcc.cc:205] Tried to move back new op lower bound from 7333327072797802496 to 7333327072628948992. Current Snapshot: MvccSnapshot[applied={T|T < 7333327072797802496}]
I20260925 19:04:14.886029 20583 mvcc.cc:205] Tried to move back new op lower bound from 7333327072797802496 to 7333327072628948992. Current Snapshot: MvccSnapshot[applied={T|T < 7333327072797802496}]
I20260925 19:04:14.886157 20582 mvcc.cc:205] Tried to move back new op lower bound from 7333327072797802496 to 7333327072628948992. Current Snapshot: MvccSnapshot[applied={T|T < 7333327072797802496}]
I20260925 19:04:14.888213 20102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:14.888293 20102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 19:04:14.889005 20102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ba4f5a202234d2abb6d06ecb243c2f3: Sending DeleteTablet for 3 replicas of tablet 2dc9c429d69343ab8d59435d09454161
I20260925 19:04:14.889314 20226 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc9c429d69343ab8d59435d09454161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:14 UTC) from {username='slave'} at 127.0.0.1:59770
I20260925 19:04:14.889420 20488 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc9c429d69343ab8d59435d09454161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:14 UTC) from {username='slave'} at 127.0.0.1:58430
I20260925 19:04:14.889513 20591 tablet_replica.cc:330] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc: stopping tablet replica
I20260925 19:04:14.889580 20357 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc9c429d69343ab8d59435d09454161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 19:04:14 UTC) from {username='slave'} at 127.0.0.1:58112
I20260925 19:04:14.889621 20591 raft_consensus.cc:2229] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:14.889720 20591 raft_consensus.cc:2258] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:14.889961 20593 tablet_replica.cc:330] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: stopping tablet replica
I20260925 19:04:14.890049 20593 raft_consensus.cc:2229] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 19:04:14.890074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 20164
I20260925 19:04:14.890137 20593 raft_consensus.cc:2258] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:14.890296 20592 tablet_replica.cc:330] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: stopping tablet replica
I20260925 19:04:14.890369 20592 raft_consensus.cc:2229] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 1 LEADER]: Raft consensus shutting down.
I20260925 19:04:14.890462 20592 raft_consensus.cc:2258] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 19:04:14.890153 20591 ts_tablet_manager.cc:1905] T 2dc9c429d69343ab8d59435d09454161 P 2e5c619a999d443db3065677b83044cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:14.890995 20593 ts_tablet_manager.cc:1905] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:14.891601 20592 ts_tablet_manager.cc:1905] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 19:04:14.892817 20593 ts_tablet_manager.cc:1918] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:14.892897 20593 log.cc:1196] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-1/wal/wals/2dc9c429d69343ab8d59435d09454161
I20260925 19:04:14.893170 20593 ts_tablet_manager.cc:1939] T 2dc9c429d69343ab8d59435d09454161 P f6202d3ca06341ba872c37be5c58f5ff: Deleting consensus metadata
I20260925 19:04:14.893288 20592 ts_tablet_manager.cc:1918] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 19:04:14.893365 20592 log.cc:1196] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: Deleting WAL directory at /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790362794046639-392-0/minicluster-data/ts-2/wal/wals/2dc9c429d69343ab8d59435d09454161
I20260925 19:04:14.893566 20090 catalog_manager.cc:4841] TS f6202d3ca06341ba872c37be5c58f5ff (127.0.98.2:36659): tablet 2dc9c429d69343ab8d59435d09454161 (table test-table [id=e3fa8a692e4048d3a0bdac317db271b0]) successfully deleted
I20260925 19:04:14.893615 20592 ts_tablet_manager.cc:1939] T 2dc9c429d69343ab8d59435d09454161 P 667946fc07024c1f8550e4251605ad43: Deleting consensus metadata
I20260925 19:04:14.894070 20090 catalog_manager.cc:4841] TS 667946fc07024c1f8550e4251605ad43 (127.0.98.3:39627): tablet 2dc9c429d69343ab8d59435d09454161 (table test-table [id=e3fa8a692e4048d3a0bdac317db271b0]) successfully deleted
W20260925 19:04:14.896610 20089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45585 (error 108)
I20260925 19:04:14.896832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 20295
I20260925 19:04:14.897399 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:14.896923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45585 (local address 127.0.0.1:59778)
0925 19:04:14.897330 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:14.897363 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390}
W20260925 19:04:14.897564 20089 catalog_manager.cc:4584] TS 2e5c619a999d443db3065677b83044cc (127.0.98.1:45585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2dc9c429d69343ab8d59435d09454161: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45585: connect: Connection refused (error 111)
I20260925 19:04:14.901816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 20426
I20260925 19:04:14.906643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 20073
2026-09-25T19:04:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (808 ms)
[----------] 6 tests from EncryptionPolicyTest (8104 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:14 dist-test-slave-49bz krb5kdc[20597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:14 dist-test-slave-49bz krb5kdc[20597](info): set up 2 sockets
Sep 25 19:04:14 dist-test-slave-49bz krb5kdc[20597](info): commencing operation
krb5kdc: starting...
W20260925 19:04:15.929996 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:16 dist-test-slave-49bz krb5kdc[20597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:16.044924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:16.125073 20610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:16.125265 20610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:16.125286 20610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:16.126392 20610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:16.126431 20610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:16.126446 20610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:16.126459 20610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:16.126472 20610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:16.127904 20610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:16.128090 20610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:16.128250 20610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:04:16.133100 20610 server_base.cc:1034] running on GCE node
W20260925 19:04:16.133131 20615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:16.133117 20616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:16.133244 20618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:16.133541 20610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:16.133816 20610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:16.134950 20610 hybrid_clock.cc:648] HybridClock initialized: now 1790363056134930 us; error 40 us; skew 500 ppm
Sep 25 19:04:16 dist-test-slave-49bz krb5kdc[20597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:16.137475 20610 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:16.137918 20610 webserver.cc:460] Webserver started at http://127.0.0.1:42517/ using document root <none> and password file <none>
I20260925 19:04:16.138134 20610 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:16.138178 20610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:16.138339 20610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:16.139413 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "eebdff4706d94a908bd0c17b0362bbce"
format_stamp: "Formatted at 2026-09-25 19:04:16 on dist-test-slave-49bz"
server_key: "0e86c8eed1c78bad1dd3a92864a8c5b9"
server_key_iv: "95844a0a894357dcf4bbea318fd0fa26"
server_key_version: "encryptionkey@0"
I20260925 19:04:16.139783 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "eebdff4706d94a908bd0c17b0362bbce"
format_stamp: "Formatted at 2026-09-25 19:04:16 on dist-test-slave-49bz"
server_key: "0e86c8eed1c78bad1dd3a92864a8c5b9"
server_key_iv: "95844a0a894357dcf4bbea318fd0fa26"
server_key_version: "encryptionkey@0"
I20260925 19:04:16.141125 20610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:16.141901 20625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:16.142132 20610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:16.142204 20610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "eebdff4706d94a908bd0c17b0362bbce"
format_stamp: "Formatted at 2026-09-25 19:04:16 on dist-test-slave-49bz"
server_key: "0e86c8eed1c78bad1dd3a92864a8c5b9"
server_key_iv: "95844a0a894357dcf4bbea318fd0fa26"
server_key_version: "encryptionkey@0"
I20260925 19:04:16.142297 20610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:16.158066 20610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:16.160080 20610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:16.160182 20610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:16.163972 20610 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43533
I20260925 19:04:16.164033 20677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43533 every 8 connection(s)
I20260925 19:04:16.164362 20610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:16.165218 20678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:16.167480 20678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce: Bootstrap starting.
I20260925 19:04:16.168041 20678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:16.168319 20678 log.cc:824] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:16.169000 20678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce: No bootstrap required, opened a new log
I20260925 19:04:16.169344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 20610
I20260925 19:04:16.169461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:16.169654 392 external_mini_cluster.cc:1428] Setting key 24ace2c4fbeda18737f983024e82ef93
I20260925 19:04:16.170917 20678 raft_consensus.cc:348] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebdff4706d94a908bd0c17b0362bbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260925 19:04:16.171085 20678 raft_consensus.cc:374] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:16.171129 20678 raft_consensus.cc:729] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebdff4706d94a908bd0c17b0362bbce, State: Initialized, Role: FOLLOWER
I20260925 19:04:16.171272 20678 consensus_queue.cc:260] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebdff4706d94a908bd0c17b0362bbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260925 19:04:16.171391 20678 raft_consensus.cc:388] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:16.171443 20678 raft_consensus.cc:482] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:16.171494 20678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:16.172293 20678 raft_consensus.cc:504] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebdff4706d94a908bd0c17b0362bbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260925 19:04:16.172415 20678 leader_election.cc:302] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eebdff4706d94a908bd0c17b0362bbce; no voters:
I20260925 19:04:16.172600 20678 leader_election.cc:288] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:16.172641 20683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:16.172747 20683 raft_consensus.cc:686] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [term 1 LEADER]: Becoming Leader. State: Replica: eebdff4706d94a908bd0c17b0362bbce, State: Running, Role: LEADER
I20260925 19:04:16.172851 20678 sys_catalog.cc:564] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:16.172824 20683 consensus_queue.cc:237] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebdff4706d94a908bd0c17b0362bbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260925 19:04:16.173159 20685 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader eebdff4706d94a908bd0c17b0362bbce. Latest consensus state: current_term: 1 leader_uuid: "eebdff4706d94a908bd0c17b0362bbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebdff4706d94a908bd0c17b0362bbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } } }
I20260925 19:04:16.173237 20685 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:16.173432 20687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:16.174124 20687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:16.174590 20684 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eebdff4706d94a908bd0c17b0362bbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebdff4706d94a908bd0c17b0362bbce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } } }
I20260925 19:04:16.174674 20684 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:16.175810 20687 catalog_manager.cc:1347] Generated new cluster ID: c7b38677b669416f8d3356b007cef347
I20260925 19:04:16.175863 20687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:04:16 dist-test-slave-49bz krb5kdc[20597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:04:16.180752 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:16.170177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37373 (local address 127.0.0.1:43533)
0925 19:04:16.170365 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:16.170375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:16.170405 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:04:16.170902 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:16.170907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:16.170952 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:16.171090 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:16.171858 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:16.172927 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:16.174064 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:16.174658 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:16.175048 (+ 390us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:16.177418 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:16.177422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:16.177422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:16.177434 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:16.179195 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:16.179601 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:16.179604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:16.179604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:16.179709 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:16.179956 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:16.179957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:16.179957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:16.180201 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:16.180402 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:16.180485 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 19:04:16.180587 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260925 19:04:16.182009 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:16.192595 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:16.186519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46635 (local address 127.0.0.1:43533)
0925 19:04:16.186553 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:16.186556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:16.186736 (+ 180us) server_negotiation.cc:407] Connection header received
0925 19:04:16.186796 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:16.186797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:16.186806 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:16.186859 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:16.187365 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:16.187870 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:16.188579 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:16.188807 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:16.188886 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:16.190030 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:16.190031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:16.190032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:16.190038 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:16.191446 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:16.191808 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:16.191809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:16.191809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:16.191871 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:16.192111 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:16.192112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:16.192112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:16.192215 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:16.192318 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:16.192435 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 19:04:16.192492 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 19:04:16.198639 20687 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:16.199175 20687 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:16.206221 20687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eebdff4706d94a908bd0c17b0362bbce: Generated new TSK 0
I20260925 19:04:16.206458 20687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:16.208349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 20610
W20260925 19:04:16.212280 20595 connection.cc:535] client connection to 127.0.0.1:43533 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43533): Connection reset by peer (error 104)
2026-09-25T19:04:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:16 dist-test-slave-49bz krb5kdc[20709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:16 dist-test-slave-49bz krb5kdc[20709](info): set up 2 sockets
Sep 25 19:04:16 dist-test-slave-49bz krb5kdc[20709](info): commencing operation
krb5kdc: starting...
W20260925 19:04:17.214180 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:17 dist-test-slave-49bz krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:17.329617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:17.406596 20722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:17.406831 20722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:17.406862 20722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:17.408033 20722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:17.408118 20722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:17.408141 20722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:17.408170 20722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:17.408200 20722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:17.409616 20722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:17.409884 20722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:17.410135 20722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:17.415098 20728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:17.415117 20727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:17.415117 20730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:17.415257 20722 server_base.cc:1034] running on GCE node
I20260925 19:04:17.415572 20722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:17.415856 20722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:17.417022 20722 hybrid_clock.cc:648] HybridClock initialized: now 1790363057417020 us; error 36 us; skew 500 ppm
Sep 25 19:04:17 dist-test-slave-49bz krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:17.426934 20722 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:17.427423 20722 webserver.cc:460] Webserver started at http://127.0.0.1:38717/ using document root <none> and password file <none>
I20260925 19:04:17.427675 20722 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:17.427744 20722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:17.427893 20722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:17.428915 20722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8"
format_stamp: "Formatted at 2026-09-25 19:04:17 on dist-test-slave-49bz"
server_key: "c80d8012553d601a1ce96a99f773a5a5"
server_key_iv: "9ac593e3e08deeb5a5b60c7741d59e7e"
server_key_version: "encryptionkey@0"
I20260925 19:04:17.429265 20722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8"
format_stamp: "Formatted at 2026-09-25 19:04:17 on dist-test-slave-49bz"
server_key: "c80d8012553d601a1ce96a99f773a5a5"
server_key_iv: "9ac593e3e08deeb5a5b60c7741d59e7e"
server_key_version: "encryptionkey@0"
I20260925 19:04:17.430639 20722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:17.431381 20737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:17.431560 20722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:17.431671 20722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8"
format_stamp: "Formatted at 2026-09-25 19:04:17 on dist-test-slave-49bz"
server_key: "c80d8012553d601a1ce96a99f773a5a5"
server_key_iv: "9ac593e3e08deeb5a5b60c7741d59e7e"
server_key_version: "encryptionkey@0"
I20260925 19:04:17.431753 20722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:17.438184 20722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:17.440469 20722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:17.440630 20722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:17.444574 20722 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40193
I20260925 19:04:17.444617 20789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40193 every 8 connection(s)
I20260925 19:04:17.444939 20722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:17.445730 20790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:17.447914 20790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8: Bootstrap starting.
I20260925 19:04:17.448503 20790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:17.448786 20790 log.cc:824] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:17.449447 20790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8: No bootstrap required, opened a new log
I20260925 19:04:17.451071 20790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } }
I20260925 19:04:17.451195 20790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:17.451215 20790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9bd2b4fa294f1691828fb2d05fdbd8, State: Initialized, Role: FOLLOWER
I20260925 19:04:17.451414 20790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } }
I20260925 19:04:17.451478 20790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:17.451524 20790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:17.451573 20790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:17.452086 20790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } }
I20260925 19:04:17.452199 20790 leader_election.cc:302] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a9bd2b4fa294f1691828fb2d05fdbd8; no voters:
I20260925 19:04:17.452378 20790 leader_election.cc:288] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:17.452436 20793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:17.452612 20793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a9bd2b4fa294f1691828fb2d05fdbd8, State: Running, Role: LEADER
I20260925 19:04:17.452633 20790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:17.452760 20793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } }
I20260925 19:04:17.453178 20794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } } }
I20260925 19:04:17.453188 20795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a9bd2b4fa294f1691828fb2d05fdbd8. Latest consensus state: current_term: 1 leader_uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9bd2b4fa294f1691828fb2d05fdbd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } } }
I20260925 19:04:17.453348 20794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:17.453372 20795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:17.453656 20797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:17.454174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 20722
I20260925 19:04:17.454358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:17.454377 20797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:17.454551 392 external_mini_cluster.cc:1428] Setting key e227aa387f174a3036c340b3dd598f8f
I20260925 19:04:17.456349 20797 catalog_manager.cc:1347] Generated new cluster ID: cf2de0bedf0047fe98d1c4ff00637c92
I20260925 19:04:17.456401 20797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:17.460436 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:17.454991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57263 (local address 127.0.0.1:40193)
0925 19:04:17.455164 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:17.455173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:17.455204 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:04:17.455315 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:17.455321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:17.455375 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:17.455549 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:17.455770 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:17.457051 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:17.458120 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:17.458766 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:17.458856 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:17.459756 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:17.459760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:17.459761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:17.459949 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:17.459985 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:17.459998 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:17.460064 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:17.460225 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 19:04:17.460248 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260925 19:04:17.461375 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:17.466347 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:17.462887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38279 (local address 127.0.0.1:40193)
0925 19:04:17.462924 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:17.462926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:17.463058 (+ 132us) server_negotiation.cc:407] Connection header received
0925 19:04:17.463102 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:17.463103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:17.463114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:17.463165 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:17.463299 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:17.464012 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:17.465058 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:17.465353 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:17.465408 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:17.466091 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:17.466092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:17.466093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:17.466096 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:17.466110 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:17.466114 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:17.466167 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:17.466261 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 19:04:17.466274 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:04:17.468818 20797 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:17.469471 20797 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:17.488001 20797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a9bd2b4fa294f1691828fb2d05fdbd8: Generated new TSK 0
I20260925 19:04:17.488191 20797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:17.494513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 20722
W20260925 19:04:17.498181 20707 connection.cc:535] client connection to 127.0.0.1:40193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40193): Connection reset by peer (error 104)
2026-09-25T19:04:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:17 dist-test-slave-49bz krb5kdc[20821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:17 dist-test-slave-49bz krb5kdc[20821](info): set up 2 sockets
Sep 25 19:04:17 dist-test-slave-49bz krb5kdc[20821](info): commencing operation
krb5kdc: starting...
W20260925 19:04:18.526324 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:18 dist-test-slave-49bz krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:18.642213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:18.718547 20834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:18.718812 20834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:18.718851 20834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:18.719974 20834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:18.720077 20834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:18.720101 20834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:18.720129 20834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:18.720156 20834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:18.721587 20834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:18.721863 20834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:18.722090 20834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:18.727181 20842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:18.727276 20834 server_base.cc:1034] running on GCE node
W20260925 19:04:18.727209 20840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:18.727243 20839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:18.727643 20834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:18.727912 20834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:18.729086 20834 hybrid_clock.cc:648] HybridClock initialized: now 1790363058729081 us; error 39 us; skew 500 ppm
Sep 25 19:04:18 dist-test-slave-49bz krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:18.731456 20834 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:18.731858 20834 webserver.cc:460] Webserver started at http://127.0.0.1:39595/ using document root <none> and password file <none>
I20260925 19:04:18.732092 20834 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:18.732162 20834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:18.732304 20834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:18.733299 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "718bd2a79c9e438ca03ce4fccbb35588"
format_stamp: "Formatted at 2026-09-25 19:04:18 on dist-test-slave-49bz"
server_key: "4acb1523f2e494a800c0b1a70ae68e9a"
server_key_iv: "513bbbc26e7beb6834bc45bcbf62b412"
server_key_version: "encryptionkey@0"
I20260925 19:04:18.733687 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "718bd2a79c9e438ca03ce4fccbb35588"
format_stamp: "Formatted at 2026-09-25 19:04:18 on dist-test-slave-49bz"
server_key: "4acb1523f2e494a800c0b1a70ae68e9a"
server_key_iv: "513bbbc26e7beb6834bc45bcbf62b412"
server_key_version: "encryptionkey@0"
I20260925 19:04:18.735075 20834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:18.735738 20849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:18.735908 20834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:18.736049 20834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "718bd2a79c9e438ca03ce4fccbb35588"
format_stamp: "Formatted at 2026-09-25 19:04:18 on dist-test-slave-49bz"
server_key: "4acb1523f2e494a800c0b1a70ae68e9a"
server_key_iv: "513bbbc26e7beb6834bc45bcbf62b412"
server_key_version: "encryptionkey@0"
I20260925 19:04:18.736102 20834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:18.739223 20834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:18.740852 20834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:18.740955 20834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:18.744649 20834 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40917
I20260925 19:04:18.744696 20901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40917 every 8 connection(s)
I20260925 19:04:18.745024 20834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:18.745950 20902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:18.746239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 20834
I20260925 19:04:18.746416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:18.746605 392 external_mini_cluster.cc:1428] Setting key 60e13f09d8cebe822aea9b8d20cca4b0
I20260925 19:04:18.748804 20902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588: Bootstrap starting.
I20260925 19:04:18.748986 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:18.747107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44153 (local address 127.0.0.1:40917)
0925 19:04:18.747311 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:18.747322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:18.747344 (+ 22us) server_negotiation.cc:407] Connection header received
0925 19:04:18.747697 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:18.747703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:18.747724 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:18.747829 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:18.748493 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:18.748496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:18.748496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:18.748500 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:18.748534 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:18.748545 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:18.748605 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:18.748703 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 19:04:18.748769 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":94,"threads_started":1}
I20260925 19:04:18.749472 20902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:18.749782 20902 log.cc:824] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:18.749800 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:18.750530 20902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588: No bootstrap required, opened a new log
I20260925 19:04:18.752303 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:18.751361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42441 (local address 127.0.0.1:40917)
0925 19:04:18.751396 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:18.751398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:18.751442 (+ 44us) server_negotiation.cc:407] Connection header received
0925 19:04:18.751517 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:18.751518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:18.751521 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:18.751565 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:18.752085 (+ 520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:18.752086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:18.752086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:18.752089 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:18.752100 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:18.752103 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:18.752147 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:18.752201 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 19:04:18.752210 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:04:18.753088 20902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718bd2a79c9e438ca03ce4fccbb35588" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260925 19:04:18.753244 20902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:18.753291 20902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718bd2a79c9e438ca03ce4fccbb35588, State: Initialized, Role: FOLLOWER
I20260925 19:04:18.753404 20902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718bd2a79c9e438ca03ce4fccbb35588" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260925 19:04:18.753510 20902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:18.753553 20902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:18.753607 20902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:18.754115 20902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718bd2a79c9e438ca03ce4fccbb35588" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260925 19:04:18.754235 20902 leader_election.cc:302] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 718bd2a79c9e438ca03ce4fccbb35588; no voters:
I20260925 19:04:18.754422 20902 leader_election.cc:288] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:18.754478 20912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:18.754657 20912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [term 1 LEADER]: Becoming Leader. State: Replica: 718bd2a79c9e438ca03ce4fccbb35588, State: Running, Role: LEADER
I20260925 19:04:18.754683 20902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:18.754786 20912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718bd2a79c9e438ca03ce4fccbb35588" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260925 19:04:18.755158 20913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "718bd2a79c9e438ca03ce4fccbb35588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718bd2a79c9e438ca03ce4fccbb35588" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } } }
I20260925 19:04:18.755167 20914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 718bd2a79c9e438ca03ce4fccbb35588. Latest consensus state: current_term: 1 leader_uuid: "718bd2a79c9e438ca03ce4fccbb35588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718bd2a79c9e438ca03ce4fccbb35588" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } } }
I20260925 19:04:18.755344 20913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:18.755353 20914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:18.755720 20921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:18.756199 20921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:18.757697 20921 catalog_manager.cc:1347] Generated new cluster ID: 81de565fe3d74794a591efdba726b425
I20260925 19:04:18.757750 20921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:18.778556 20921 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:18.778743 20921 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:18.783972 20921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 718bd2a79c9e438ca03ce4fccbb35588: Generated new TSK 0
I20260925 19:04:18.784121 20921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:18.790940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 20834
2026-09-25T19:04:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:18 dist-test-slave-49bz krb5kdc[20933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:18 dist-test-slave-49bz krb5kdc[20933](info): set up 2 sockets
Sep 25 19:04:18 dist-test-slave-49bz krb5kdc[20933](info): commencing operation
krb5kdc: starting...
W20260925 19:04:19.814116 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:19 dist-test-slave-49bz krb5kdc[20933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:19.933110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:20.012149 20946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:20.012342 20946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:20.012364 20946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:20.013505 20946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:20.013548 20946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:20.013562 20946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:20.013576 20946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:20.013630 20946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:20.015105 20946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:20.015407 20946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:20.015695 20946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:20.020625 20954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:20.020653 20946 server_base.cc:1034] running on GCE node
W20260925 19:04:20.020625 20952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:20.020676 20951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:20.021104 20946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:20.021348 20946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:20.022485 20946 hybrid_clock.cc:648] HybridClock initialized: now 1790363060022475 us; error 31 us; skew 500 ppm
Sep 25 19:04:20 dist-test-slave-49bz krb5kdc[20933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:20.024873 20946 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:20.025326 20946 webserver.cc:460] Webserver started at http://127.0.0.1:40857/ using document root <none> and password file <none>
I20260925 19:04:20.025499 20946 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:20.025537 20946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:20.025619 20946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:20.026706 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "ce017f99674c4987ac0f9b07bdab70ee"
format_stamp: "Formatted at 2026-09-25 19:04:20 on dist-test-slave-49bz"
server_key: "2384aff8cc63aa0b2f428a7d20abc864"
server_key_iv: "da1bf04af62b1f8d18ac04e79a02e675"
server_key_version: "encryptionkey@0"
I20260925 19:04:20.027001 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce017f99674c4987ac0f9b07bdab70ee"
format_stamp: "Formatted at 2026-09-25 19:04:20 on dist-test-slave-49bz"
server_key: "2384aff8cc63aa0b2f428a7d20abc864"
server_key_iv: "da1bf04af62b1f8d18ac04e79a02e675"
server_key_version: "encryptionkey@0"
I20260925 19:04:20.028218 20946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:20.028906 20961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:20.029129 20946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:20.029214 20946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "ce017f99674c4987ac0f9b07bdab70ee"
format_stamp: "Formatted at 2026-09-25 19:04:20 on dist-test-slave-49bz"
server_key: "2384aff8cc63aa0b2f428a7d20abc864"
server_key_iv: "da1bf04af62b1f8d18ac04e79a02e675"
server_key_version: "encryptionkey@0"
I20260925 19:04:20.029261 20946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:20.047717 20946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:20.050036 20946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:20.050144 20946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:20.053880 20946 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40441
I20260925 19:04:20.053936 21013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40441 every 8 connection(s)
I20260925 19:04:20.054278 20946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:20.055153 21014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:20.057318 21014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee: Bootstrap starting.
I20260925 19:04:20.057811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 20946
I20260925 19:04:20.057922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:20.057978 21014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:20.058135 392 external_mini_cluster.cc:1428] Setting key 09ae85d2e64980210568a0570a81e24e
I20260925 19:04:20.058333 21014 log.cc:824] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:20.059178 21014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee: No bootstrap required, opened a new log
I20260925 19:04:20.061407 21014 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce017f99674c4987ac0f9b07bdab70ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40441 } }
I20260925 19:04:20.061573 21014 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:20.061607 21014 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce017f99674c4987ac0f9b07bdab70ee, State: Initialized, Role: FOLLOWER
I20260925 19:04:20.061719 21014 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce017f99674c4987ac0f9b07bdab70ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40441 } }
I20260925 19:04:20.061787 21014 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:20.061813 21014 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:20.061854 21014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:20.062495 21014 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce017f99674c4987ac0f9b07bdab70ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40441 } }
I20260925 19:04:20.062616 21014 leader_election.cc:302] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce017f99674c4987ac0f9b07bdab70ee; no voters:
I20260925 19:04:20.062819 21014 leader_election.cc:288] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:20.062970 21020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:20.063072 21014 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:20.063455 21020 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [term 1 LEADER]: Becoming Leader. State: Replica: ce017f99674c4987ac0f9b07bdab70ee, State: Running, Role: LEADER
I20260925 19:04:20.063588 21020 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce017f99674c4987ac0f9b07bdab70ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40441 } }
I20260925 19:04:20.063941 21019 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce017f99674c4987ac0f9b07bdab70ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce017f99674c4987ac0f9b07bdab70ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40441 } } }
I20260925 19:04:20.064015 21019 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:20.064158 21021 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce017f99674c4987ac0f9b07bdab70ee. Latest consensus state: current_term: 1 leader_uuid: "ce017f99674c4987ac0f9b07bdab70ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce017f99674c4987ac0f9b07bdab70ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40441 } } }
I20260925 19:04:20.064215 21021 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:20.064718 21028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 19:04:20 dist-test-slave-49bz krb5kdc[20933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:04:20.065343 21028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:20.066920 21028 catalog_manager.cc:1347] Generated new cluster ID: 7e898eee41e04a3694a3a5c75b493b62
I20260925 19:04:20.066968 21028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:20.072834 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:20.058689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43819 (local address 127.0.0.1:40441)
0925 19:04:20.058866 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:20.058876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:20.058912 (+ 36us) server_negotiation.cc:407] Connection header received
0925 19:04:20.059405 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:20.059412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:20.059459 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:20.059571 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:20.060374 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:20.061199 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:20.062236 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:20.062743 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:20.062839 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:20.065748 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:20.065751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:20.065752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:20.065763 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:20.071241 (+ 5478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:20.071660 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:20.071662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:20.071663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:20.071758 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:20.071974 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:20.071974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:20.071975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:20.072247 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:20.072388 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:20.072513 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 19:04:20.072633 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260925 19:04:20.073762 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:20.079129 21028 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:20.079643 21028 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:20.082168 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:20.075268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35415 (local address 127.0.0.1:40441)
0925 19:04:20.075297 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:20.075299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:20.075349 (+ 50us) server_negotiation.cc:407] Connection header received
0925 19:04:20.075417 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:20.075418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:20.075426 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:20.075479 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:20.076100 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:20.076770 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:20.077769 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:20.078075 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:20.078133 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:20.079641 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:20.079642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:20.079642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:20.079648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:20.081106 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:20.081459 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:20.081459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:20.081460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:20.081508 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:20.081697 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:20.081698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:20.081698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:20.081819 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:20.081898 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:20.082020 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 19:04:20.082093 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 19:04:20.083916 21028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce017f99674c4987ac0f9b07bdab70ee: Generated new TSK 0
I20260925 19:04:20.084070 21028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:20.091367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 20946
2026-09-25T19:04:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:20 dist-test-slave-49bz krb5kdc[21045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:20 dist-test-slave-49bz krb5kdc[21045](info): set up 2 sockets
Sep 25 19:04:20 dist-test-slave-49bz krb5kdc[21045](info): commencing operation
krb5kdc: starting...
W20260925 19:04:21.110404 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:21 dist-test-slave-49bz krb5kdc[21045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:21.229662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:21.309021 21058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:21.309224 21058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:21.309247 21058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:21.310461 21058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:21.310501 21058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:21.310515 21058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:21.310527 21058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:21.310540 21058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:21.312055 21058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:21.312310 21058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:21.312532 21058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:21.317310 21066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:21.317436 21058 server_base.cc:1034] running on GCE node
W20260925 19:04:21.317302 21063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:21.317304 21064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:21.317734 21058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:21.318043 21058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:21.319212 21058 hybrid_clock.cc:648] HybridClock initialized: now 1790363061319175 us; error 48 us; skew 500 ppm
Sep 25 19:04:21 dist-test-slave-49bz krb5kdc[21045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:21.321638 21058 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:21.321992 21058 webserver.cc:460] Webserver started at http://127.0.0.1:44685/ using document root <none> and password file <none>
I20260925 19:04:21.322188 21058 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:21.322228 21058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:21.322389 21058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:21.323359 21058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "08fac3eeb5944a74b1b279f1ac1e6dca"
format_stamp: "Formatted at 2026-09-25 19:04:21 on dist-test-slave-49bz"
server_key: "050183770372be2324686f047f894f64"
server_key_iv: "2194cb517f56ccea2e6c3fe11aa860d5"
server_key_version: "encryptionkey@0"
I20260925 19:04:21.323770 21058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "08fac3eeb5944a74b1b279f1ac1e6dca"
format_stamp: "Formatted at 2026-09-25 19:04:21 on dist-test-slave-49bz"
server_key: "050183770372be2324686f047f894f64"
server_key_iv: "2194cb517f56ccea2e6c3fe11aa860d5"
server_key_version: "encryptionkey@0"
I20260925 19:04:21.325011 21058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:21.325665 21073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:21.325831 21058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:21.325932 21058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "08fac3eeb5944a74b1b279f1ac1e6dca"
format_stamp: "Formatted at 2026-09-25 19:04:21 on dist-test-slave-49bz"
server_key: "050183770372be2324686f047f894f64"
server_key_iv: "2194cb517f56ccea2e6c3fe11aa860d5"
server_key_version: "encryptionkey@0"
I20260925 19:04:21.326013 21058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:21.343964 21058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:21.345729 21058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:21.345889 21058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:21.349822 21058 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37667
I20260925 19:04:21.349866 21125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37667 every 8 connection(s)
I20260925 19:04:21.350221 21058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:21.351091 21126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:21.353217 21126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca: Bootstrap starting.
I20260925 19:04:21.353794 21126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:21.354099 21126 log.cc:824] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:21.354163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 21058
I20260925 19:04:21.354310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:21.354517 392 external_mini_cluster.cc:1428] Setting key 2f2ba95d295894090e42452e55a3654e
I20260925 19:04:21.354856 21126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca: No bootstrap required, opened a new log
I20260925 19:04:21.356550 21126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fac3eeb5944a74b1b279f1ac1e6dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260925 19:04:21.356711 21126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:21.356748 21126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fac3eeb5944a74b1b279f1ac1e6dca, State: Initialized, Role: FOLLOWER
I20260925 19:04:21.356935 21126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fac3eeb5944a74b1b279f1ac1e6dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260925 19:04:21.357048 21126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:21.357089 21126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:21.357131 21126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:21.357610 21126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fac3eeb5944a74b1b279f1ac1e6dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260925 19:04:21.357723 21126 leader_election.cc:302] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08fac3eeb5944a74b1b279f1ac1e6dca; no voters:
I20260925 19:04:21.357903 21126 leader_election.cc:288] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:21.357926 21131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:21.358038 21131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [term 1 LEADER]: Becoming Leader. State: Replica: 08fac3eeb5944a74b1b279f1ac1e6dca, State: Running, Role: LEADER
I20260925 19:04:21.358168 21126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:21.358186 21131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fac3eeb5944a74b1b279f1ac1e6dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } }
I20260925 19:04:21.359794 21132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08fac3eeb5944a74b1b279f1ac1e6dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fac3eeb5944a74b1b279f1ac1e6dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } } }
I20260925 19:04:21.359823 21133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08fac3eeb5944a74b1b279f1ac1e6dca. Latest consensus state: current_term: 1 leader_uuid: "08fac3eeb5944a74b1b279f1ac1e6dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fac3eeb5944a74b1b279f1ac1e6dca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37667 } } }
W20260925 19:04:21.359913 21146 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:04:21.359951 21146 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:04:21.359946 21133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:21.360127 21132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:21.360253 21148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:21.360419 21148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:21.361862 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:21.355061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41691 (local address 127.0.0.1:37667)
0925 19:04:21.355240 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:21.355247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:21.355270 (+ 23us) server_negotiation.cc:407] Connection header received
0925 19:04:21.356543 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:21.356549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:21.356594 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:21.356703 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:21.356969 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:21.358398 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:21.359503 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:21.360129 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:21.360235 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:21.361107 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:21.361111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:21.361111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:21.361319 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:21.361355 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:21.361370 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:21.361441 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:21.361560 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 19:04:21.361626 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260925 19:04:21.362510 21148 catalog_manager.cc:1347] Generated new cluster ID: 85f5398b525f4f4b9123ebd35ffb7324
I20260925 19:04:21.362558 21148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:21.362828 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:21.369807 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:21.366522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34975 (local address 127.0.0.1:37667)
0925 19:04:21.366567 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:21.366568 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:21.366719 (+ 151us) server_negotiation.cc:407] Connection header received
0925 19:04:21.366742 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:21.366743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:21.366764 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:21.366822 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:21.366923 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:21.367593 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:21.368572 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:21.368860 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:21.368932 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:21.369580 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:21.369581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:21.369581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:21.369585 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:21.369599 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:21.369603 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:21.369660 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:21.369719 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 19:04:21.369741 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:04:21.371771 21148 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:21.372257 21148 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:21.377722 21148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08fac3eeb5944a74b1b279f1ac1e6dca: Generated new TSK 0
I20260925 19:04:21.377938 21148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:21.380769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 21058
2026-09-25T19:04:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:21 dist-test-slave-49bz krb5kdc[21157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:21 dist-test-slave-49bz krb5kdc[21157](info): set up 2 sockets
Sep 25 19:04:21 dist-test-slave-49bz krb5kdc[21157](info): commencing operation
krb5kdc: starting...
W20260925 19:04:22.414314 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:22 dist-test-slave-49bz krb5kdc[21157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:22.535073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:22.615432 21170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:22.615621 21170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:22.615643 21170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:22.616709 21170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:22.616740 21170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:22.616755 21170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:22.616765 21170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:22.616777 21170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:22.618147 21170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:22.618403 21170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:22.618577 21170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:22.623224 21175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:22.623224 21178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:22.623330 21176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:22.623648 21170 server_base.cc:1034] running on GCE node
I20260925 19:04:22.623860 21170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:22.624085 21170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:22.625209 21170 hybrid_clock.cc:648] HybridClock initialized: now 1790363062625195 us; error 27 us; skew 500 ppm
Sep 25 19:04:22 dist-test-slave-49bz krb5kdc[21157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:22.631176 21170 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:22.631559 21170 webserver.cc:460] Webserver started at http://127.0.0.1:39827/ using document root <none> and password file <none>
I20260925 19:04:22.631810 21170 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:22.631882 21170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:22.632032 21170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:22.633062 21170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "fe1ba24bf553491e8b96df3b5ae0312f"
format_stamp: "Formatted at 2026-09-25 19:04:22 on dist-test-slave-49bz"
server_key: "aecc93c5c61f42b4bca8b099e0c969f9"
server_key_iv: "ba02db28f371eb05a5e1701fdf27d0d5"
server_key_version: "encryptionkey@0"
I20260925 19:04:22.633436 21170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe1ba24bf553491e8b96df3b5ae0312f"
format_stamp: "Formatted at 2026-09-25 19:04:22 on dist-test-slave-49bz"
server_key: "aecc93c5c61f42b4bca8b099e0c969f9"
server_key_iv: "ba02db28f371eb05a5e1701fdf27d0d5"
server_key_version: "encryptionkey@0"
I20260925 19:04:22.634838 21170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 19:04:22.635578 21185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:22.635800 21170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:22.635878 21170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "fe1ba24bf553491e8b96df3b5ae0312f"
format_stamp: "Formatted at 2026-09-25 19:04:22 on dist-test-slave-49bz"
server_key: "aecc93c5c61f42b4bca8b099e0c969f9"
server_key_iv: "ba02db28f371eb05a5e1701fdf27d0d5"
server_key_version: "encryptionkey@0"
I20260925 19:04:22.635928 21170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:22.639114 21170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:22.641321 21170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:22.641423 21170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:22.645264 21170 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38131
I20260925 19:04:22.645292 21237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38131 every 8 connection(s)
I20260925 19:04:22.645614 21170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:22.646641 21238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:22.648773 21238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f: Bootstrap starting.
I20260925 19:04:22.649329 21238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:22.649582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 21170
I20260925 19:04:22.649605 21238 log.cc:824] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:22.649704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:22.649896 392 external_mini_cluster.cc:1428] Setting key 84e6b9efec35689e96829ab3cae343d3
I20260925 19:04:22.650552 21238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f: No bootstrap required, opened a new log
I20260925 19:04:22.652253 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:22.650415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48507 (local address 127.0.0.1:38131)
0925 19:04:22.650567 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:22.650575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:22.650635 (+ 60us) server_negotiation.cc:407] Connection header received
0925 19:04:22.651053 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:22.651059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:22.651081 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:22.651198 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:22.651774 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:22.651777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:22.651778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:22.651781 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:22.651813 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:22.651834 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:22.651902 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:22.652012 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 19:04:22.652079 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260925 19:04:22.653168 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:22.653415 21238 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ba24bf553491e8b96df3b5ae0312f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38131 } }
I20260925 19:04:22.653535 21238 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:22.653556 21238 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1ba24bf553491e8b96df3b5ae0312f, State: Initialized, Role: FOLLOWER
I20260925 19:04:22.653676 21238 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ba24bf553491e8b96df3b5ae0312f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38131 } }
I20260925 19:04:22.653755 21238 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:22.653811 21238 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:22.653861 21238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:22.654448 21238 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ba24bf553491e8b96df3b5ae0312f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38131 } }
I20260925 19:04:22.654559 21238 leader_election.cc:302] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe1ba24bf553491e8b96df3b5ae0312f; no voters:
I20260925 19:04:22.654704 21238 leader_election.cc:288] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:22.654862 21249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:22.654953 21238 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:22.655031 21249 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [term 1 LEADER]: Becoming Leader. State: Replica: fe1ba24bf553491e8b96df3b5ae0312f, State: Running, Role: LEADER
I20260925 19:04:22.655120 21249 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ba24bf553491e8b96df3b5ae0312f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38131 } }
I20260925 19:04:22.655440 21249 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe1ba24bf553491e8b96df3b5ae0312f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ba24bf553491e8b96df3b5ae0312f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38131 } } }
I20260925 19:04:22.655545 21249 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:22.655685 21249 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe1ba24bf553491e8b96df3b5ae0312f. Latest consensus state: current_term: 1 leader_uuid: "fe1ba24bf553491e8b96df3b5ae0312f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ba24bf553491e8b96df3b5ae0312f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38131 } } }
I20260925 19:04:22.655781 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:22.654669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35057 (local address 127.0.0.1:38131)
0925 19:04:22.654708 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:22.654710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:22.654830 (+ 120us) server_negotiation.cc:407] Connection header received
0925 19:04:22.654893 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:22.654894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:22.654898 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:22.654953 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:22.655502 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:22.655504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:22.655504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:22.655509 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:22.655522 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:22.655527 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:22.655575 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:22.655717 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 19:04:22.655727 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 19:04:22.655926 21257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:22.655787 21249 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:22.656606 21257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:22.658169 21257 catalog_manager.cc:1347] Generated new cluster ID: 9775d7e3d29741b8b8c25c4e25050b5f
I20260925 19:04:22.658212 21257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:22.671264 21257 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:22.671447 21257 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:22.673204 21257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe1ba24bf553491e8b96df3b5ae0312f: Generated new TSK 0
I20260925 19:04:22.673365 21257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:22.681118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 21170
2026-09-25T19:04:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:22 dist-test-slave-49bz krb5kdc[21269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:22 dist-test-slave-49bz krb5kdc[21269](info): set up 2 sockets
Sep 25 19:04:22 dist-test-slave-49bz krb5kdc[21269](info): commencing operation
krb5kdc: starting...
W20260925 19:04:23.694525 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:23 dist-test-slave-49bz krb5kdc[21269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:23.814021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:23.895949 21282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:23.896157 21282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:23.896180 21282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:23.897312 21282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:23.897353 21282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:23.897367 21282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:23.897379 21282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:23.897433 21282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:23.898999 21282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:23.899255 21282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:23.899487 21282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:23.904266 21288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:23.904253 21290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:23.904381 21282 server_base.cc:1034] running on GCE node
W20260925 19:04:23.904239 21287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:23.904680 21282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:23.904920 21282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:23.906097 21282 hybrid_clock.cc:648] HybridClock initialized: now 1790363063906091 us; error 66 us; skew 500 ppm
Sep 25 19:04:23 dist-test-slave-49bz krb5kdc[21269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:23.908584 21282 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:23.909035 21282 webserver.cc:460] Webserver started at http://127.0.0.1:40539/ using document root <none> and password file <none>
I20260925 19:04:23.909268 21282 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:23.909312 21282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:23.909454 21282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:23.910539 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "b60e190716c3483ea5f9f2e81906df9e"
format_stamp: "Formatted at 2026-09-25 19:04:23 on dist-test-slave-49bz"
server_key: "233e2c33d3e6ff068fe5028fc6a1681f"
server_key_iv: "1e4456c78b2871dc3344ffd829a9bcc7"
server_key_version: "encryptionkey@0"
I20260925 19:04:23.910882 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "b60e190716c3483ea5f9f2e81906df9e"
format_stamp: "Formatted at 2026-09-25 19:04:23 on dist-test-slave-49bz"
server_key: "233e2c33d3e6ff068fe5028fc6a1681f"
server_key_iv: "1e4456c78b2871dc3344ffd829a9bcc7"
server_key_version: "encryptionkey@0"
I20260925 19:04:23.912187 21282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:23.913182 21297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:23.913415 21282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 19:04:23.913504 21282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "b60e190716c3483ea5f9f2e81906df9e"
format_stamp: "Formatted at 2026-09-25 19:04:23 on dist-test-slave-49bz"
server_key: "233e2c33d3e6ff068fe5028fc6a1681f"
server_key_iv: "1e4456c78b2871dc3344ffd829a9bcc7"
server_key_version: "encryptionkey@0"
I20260925 19:04:23.913573 21282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:23.932601 21282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:23.934571 21282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:23.934764 21282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:23.938704 21282 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45553
I20260925 19:04:23.938737 21349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45553 every 8 connection(s)
I20260925 19:04:23.939088 21282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:23.939937 21350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:23.942166 21350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e: Bootstrap starting.
I20260925 19:04:23.942766 21350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:23.943073 21350 log.cc:824] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:23.943751 21350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e: No bootstrap required, opened a new log
I20260925 19:04:23.945253 21350 raft_consensus.cc:348] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60e190716c3483ea5f9f2e81906df9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45553 } }
I20260925 19:04:23.945382 21350 raft_consensus.cc:374] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:23.945405 21350 raft_consensus.cc:729] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60e190716c3483ea5f9f2e81906df9e, State: Initialized, Role: FOLLOWER
I20260925 19:04:23.945497 21350 consensus_queue.cc:260] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60e190716c3483ea5f9f2e81906df9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45553 } }
I20260925 19:04:23.945592 21350 raft_consensus.cc:388] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:23.945621 21350 raft_consensus.cc:482] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:23.945683 21350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:23.946205 21350 raft_consensus.cc:504] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60e190716c3483ea5f9f2e81906df9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45553 } }
I20260925 19:04:23.946347 21350 leader_election.cc:302] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b60e190716c3483ea5f9f2e81906df9e; no voters:
I20260925 19:04:23.946522 21350 leader_election.cc:288] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:23.946564 21353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:23.946748 21353 raft_consensus.cc:686] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [term 1 LEADER]: Becoming Leader. State: Replica: b60e190716c3483ea5f9f2e81906df9e, State: Running, Role: LEADER
I20260925 19:04:23.946771 21350 sys_catalog.cc:564] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:23.946832 21353 consensus_queue.cc:237] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60e190716c3483ea5f9f2e81906df9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45553 } }
I20260925 19:04:23.947222 21354 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b60e190716c3483ea5f9f2e81906df9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60e190716c3483ea5f9f2e81906df9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45553 } } }
I20260925 19:04:23.947223 21355 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b60e190716c3483ea5f9f2e81906df9e. Latest consensus state: current_term: 1 leader_uuid: "b60e190716c3483ea5f9f2e81906df9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60e190716c3483ea5f9f2e81906df9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45553 } } }
I20260925 19:04:23.947355 21354 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:23.947371 21355 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:23.947850 21358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:23.948371 21358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:23.948810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 21282
I20260925 19:04:23.948935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:23.949092 392 external_mini_cluster.cc:1428] Setting key 09140619f9ccd52ca5cf28a5ec8b4235
I20260925 19:04:23.950371 21358 catalog_manager.cc:1347] Generated new cluster ID: 855ccf0c9b1744f6b19eae5b66e025d5
I20260925 19:04:23.950420 21358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:04:23 dist-test-slave-49bz krb5kdc[21269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:04:23.959889 21358 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:23.960561 21358 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:23.963685 21358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b60e190716c3483ea5f9f2e81906df9e: Generated new TSK 0
I20260925 19:04:23.963763 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:23.949655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:47571 (local address 127.0.0.1:45553)
0925 19:04:23.950048 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:23.950057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:23.950087 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:04:23.950206 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:23.950213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:23.950300 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:23.954460 (+ 4160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:23.955101 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:23.956314 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:23.957121 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:23.957558 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:23.957692 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:23.960280 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:23.960285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:23.960285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:23.960297 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:23.962027 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:23.962502 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:23.962505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:23.962505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:23.962596 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:23.962831 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:23.962832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:23.962832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:23.963084 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:23.963247 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:23.963373 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 19:04:23.963474 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":79,"threads_started":1}
I20260925 19:04:23.963830 21358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:23.964793 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:23.972812 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:23.966315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:51597 (local address 127.0.0.1:45553)
0925 19:04:23.966354 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:23.966355 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:23.966442 (+ 87us) server_negotiation.cc:407] Connection header received
0925 19:04:23.966496 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:23.966496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:23.966504 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:23.966542 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:23.967123 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:23.967585 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:23.968576 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:23.968982 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:23.969034 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:23.970349 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:23.970350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:23.970351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:23.970357 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:23.971756 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:23.972151 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:23.972151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:23.972152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:23.972198 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:23.972414 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:23.972414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:23.972415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:23.972500 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:23.972570 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:23.972712 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 19:04:23.972757 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:04:23.975102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 21282
W20260925 19:04:23.979271 21267 connection.cc:535] client connection to 127.0.0.1:45553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45553): Connection reset by peer (error 104)
2026-09-25T19:04:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:24 dist-test-slave-49bz krb5kdc[21381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:24 dist-test-slave-49bz krb5kdc[21381](info): set up 2 sockets
Sep 25 19:04:24 dist-test-slave-49bz krb5kdc[21381](info): commencing operation
krb5kdc: starting...
W20260925 19:04:25.006469 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:25 dist-test-slave-49bz krb5kdc[21381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:25.126081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:25.207222 21394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:25.207420 21394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:25.207441 21394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:25.208508 21394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:25.208549 21394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:25.208562 21394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:25.208575 21394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:25.208587 21394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:25.209950 21394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:25.210183 21394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:25.210419 21394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:25.215121 21399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:25.215116 21402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:25.215118 21400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:25.215238 21394 server_base.cc:1034] running on GCE node
I20260925 19:04:25.215608 21394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:25.215895 21394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:25.217036 21394 hybrid_clock.cc:648] HybridClock initialized: now 1790363065217029 us; error 30 us; skew 500 ppm
Sep 25 19:04:25 dist-test-slave-49bz krb5kdc[21381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:25.219488 21394 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:25.219962 21394 webserver.cc:460] Webserver started at http://127.0.0.1:38727/ using document root <none> and password file <none>
I20260925 19:04:25.220184 21394 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:25.220266 21394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:25.220372 21394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:25.221380 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165"
format_stamp: "Formatted at 2026-09-25 19:04:25 on dist-test-slave-49bz"
server_key: "fc809e17e9f4a69e1b622dbf14b07b92"
server_key_iv: "54d981eb8019639f4477498f23dba48c"
server_key_version: "encryptionkey@0"
I20260925 19:04:25.221678 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165"
format_stamp: "Formatted at 2026-09-25 19:04:25 on dist-test-slave-49bz"
server_key: "fc809e17e9f4a69e1b622dbf14b07b92"
server_key_iv: "54d981eb8019639f4477498f23dba48c"
server_key_version: "encryptionkey@0"
I20260925 19:04:25.223022 21394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:25.223693 21409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:25.223908 21394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:25.223999 21394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165"
format_stamp: "Formatted at 2026-09-25 19:04:25 on dist-test-slave-49bz"
server_key: "fc809e17e9f4a69e1b622dbf14b07b92"
server_key_iv: "54d981eb8019639f4477498f23dba48c"
server_key_version: "encryptionkey@0"
I20260925 19:04:25.224057 21394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:25.230803 21394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:25.232558 21394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:25.232663 21394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:25.236398 21394 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38767
I20260925 19:04:25.236697 21461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38767 every 8 connection(s)
I20260925 19:04:25.236758 21394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:25.237411 21462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:25.239661 21462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165: Bootstrap starting.
I20260925 19:04:25.240217 21462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:25.240528 21462 log.cc:824] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:25.240594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 21394
I20260925 19:04:25.240763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:25.240947 392 external_mini_cluster.cc:1428] Setting key d6aab43dc3de8cb4314807953e9a51b8
I20260925 19:04:25.241340 21462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165: No bootstrap required, opened a new log
I20260925 19:04:25.243444 21462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38767 } }
I20260925 19:04:25.243649 21462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:25.243685 21462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dff78a1bc1d4b9cb19f3c76ac3cd165, State: Initialized, Role: FOLLOWER
I20260925 19:04:25.243806 21462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38767 } }
I20260925 19:04:25.243892 21462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:25.243924 21462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:25.243970 21462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:25.244495 21462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38767 } }
I20260925 19:04:25.244577 21462 leader_election.cc:302] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dff78a1bc1d4b9cb19f3c76ac3cd165; no voters:
I20260925 19:04:25.244813 21462 leader_election.cc:288] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:25.244840 21467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:25.244958 21467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [term 1 LEADER]: Becoming Leader. State: Replica: 4dff78a1bc1d4b9cb19f3c76ac3cd165, State: Running, Role: LEADER
I20260925 19:04:25.245062 21462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:25.245046 21467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38767 } }
I20260925 19:04:25.245390 21467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38767 } } }
I20260925 19:04:25.245468 21467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:25.245616 21467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dff78a1bc1d4b9cb19f3c76ac3cd165. Latest consensus state: current_term: 1 leader_uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff78a1bc1d4b9cb19f3c76ac3cd165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38767 } } }
I20260925 19:04:25.245683 21467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:25.245805 21473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:25.246335 21473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:25.247993 21473 catalog_manager.cc:1347] Generated new cluster ID: ef1ddb75edc94a4c9cfbaf9a59691338
I20260925 19:04:25.248066 21473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:25.248795 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:25.241455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:40413 (local address 127.0.0.1:38767)
0925 19:04:25.241638 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:25.241646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:25.241707 (+ 61us) server_negotiation.cc:407] Connection header received
0925 19:04:25.243657 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:25.243666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:25.243733 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:25.243891 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:25.244148 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:25.245375 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:25.246769 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:25.247267 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:25.247406 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:25.248109 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:25.248113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:25.248113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:25.248292 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:25.248327 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:25.248339 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:25.248433 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:25.248568 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 19:04:25.248599 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260925 19:04:25.249703 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:25.257174 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:25.251105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:39279 (local address 127.0.0.1:38767)
0925 19:04:25.251142 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:25.251144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:25.254447 (+ 3303us) server_negotiation.cc:407] Connection header received
0925 19:04:25.254519 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:25.254520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:25.254531 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:25.254585 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:25.254745 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:25.255197 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:25.255922 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:25.256128 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:25.256207 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:25.256912 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:25.256913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:25.256914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:25.256918 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:25.256931 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:25.256935 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:25.256988 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:25.257048 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 19:04:25.257057 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:04:25.260002 21473 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:25.260650 21473 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:25.267302 21473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dff78a1bc1d4b9cb19f3c76ac3cd165: Generated new TSK 0
I20260925 19:04:25.267467 21473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:25.273775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 21394
W20260925 19:04:25.278340 21379 connection.cc:535] client connection to 127.0.0.1:38767 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38767): Connection reset by peer (error 104)
2026-09-25T19:04:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:25 dist-test-slave-49bz krb5kdc[21493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:25 dist-test-slave-49bz krb5kdc[21493](info): set up 2 sockets
Sep 25 19:04:25 dist-test-slave-49bz krb5kdc[21493](info): commencing operation
krb5kdc: starting...
W20260925 19:04:26.298184 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:26 dist-test-slave-49bz krb5kdc[21493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:26.413949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:26.494292 21506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:26.494493 21506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:26.494516 21506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:26.495603 21506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:26.495642 21506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:26.495656 21506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:26.495669 21506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:26.495682 21506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:26.497035 21506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:26.497226 21506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:26.497385 21506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:26.502290 21511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:26.502338 21514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:26.502490 21506 server_base.cc:1034] running on GCE node
W20260925 19:04:26.502344 21512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:26.502830 21506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:26.503151 21506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:26.504323 21506 hybrid_clock.cc:648] HybridClock initialized: now 1790363066504307 us; error 27 us; skew 500 ppm
Sep 25 19:04:26 dist-test-slave-49bz krb5kdc[21493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:26.506722 21506 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:26.507133 21506 webserver.cc:460] Webserver started at http://127.0.0.1:34569/ using document root <none> and password file <none>
I20260925 19:04:26.507354 21506 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:26.507424 21506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:26.507576 21506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:26.508579 21506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "adac13120cf74bcd89709fd9fec9c67a"
format_stamp: "Formatted at 2026-09-25 19:04:26 on dist-test-slave-49bz"
server_key: "7131c38fffe6fd8207de2ac2eaa53504"
server_key_iv: "cf6354d2a780cbdb73b4509a28fd38a0"
server_key_version: "encryptionkey@0"
I20260925 19:04:26.508939 21506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "adac13120cf74bcd89709fd9fec9c67a"
format_stamp: "Formatted at 2026-09-25 19:04:26 on dist-test-slave-49bz"
server_key: "7131c38fffe6fd8207de2ac2eaa53504"
server_key_iv: "cf6354d2a780cbdb73b4509a28fd38a0"
server_key_version: "encryptionkey@0"
I20260925 19:04:26.510169 21506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:04:26.510865 21521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:26.511032 21506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:26.511130 21506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "adac13120cf74bcd89709fd9fec9c67a"
format_stamp: "Formatted at 2026-09-25 19:04:26 on dist-test-slave-49bz"
server_key: "7131c38fffe6fd8207de2ac2eaa53504"
server_key_iv: "cf6354d2a780cbdb73b4509a28fd38a0"
server_key_version: "encryptionkey@0"
I20260925 19:04:26.511214 21506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:26.514787 21506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:26.516536 21506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:26.516706 21506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:26.520548 21506 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40201
I20260925 19:04:26.520577 21573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40201 every 8 connection(s)
I20260925 19:04:26.520905 21506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:26.521862 21574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:26.524080 21574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a: Bootstrap starting.
I20260925 19:04:26.524581 21574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:26.524811 21574 log.cc:824] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:26.525460 21574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a: No bootstrap required, opened a new log
I20260925 19:04:26.527061 21574 raft_consensus.cc:348] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adac13120cf74bcd89709fd9fec9c67a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40201 } }
I20260925 19:04:26.527196 21574 raft_consensus.cc:374] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:26.527220 21574 raft_consensus.cc:729] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adac13120cf74bcd89709fd9fec9c67a, State: Initialized, Role: FOLLOWER
I20260925 19:04:26.527361 21574 consensus_queue.cc:260] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adac13120cf74bcd89709fd9fec9c67a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40201 } }
I20260925 19:04:26.527443 21574 raft_consensus.cc:388] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:26.527467 21574 raft_consensus.cc:482] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:26.527493 21574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:26.527976 21574 raft_consensus.cc:504] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adac13120cf74bcd89709fd9fec9c67a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40201 } }
I20260925 19:04:26.528059 21574 leader_election.cc:302] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adac13120cf74bcd89709fd9fec9c67a; no voters:
I20260925 19:04:26.528179 21574 leader_election.cc:288] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:26.528332 21577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:26.528438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 21506
I20260925 19:04:26.528470 21574 sys_catalog.cc:564] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:26.528566 21577 raft_consensus.cc:686] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [term 1 LEADER]: Becoming Leader. State: Replica: adac13120cf74bcd89709fd9fec9c67a, State: Running, Role: LEADER
I20260925 19:04:26.528604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:26.528653 21577 consensus_queue.cc:237] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adac13120cf74bcd89709fd9fec9c67a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40201 } }
I20260925 19:04:26.528801 392 external_mini_cluster.cc:1428] Setting key 5b1be9a5d5ccd7a82df400e8c08f1f2e
I20260925 19:04:26.529253 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adac13120cf74bcd89709fd9fec9c67a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adac13120cf74bcd89709fd9fec9c67a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40201 } } }
I20260925 19:04:26.529345 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:26.529552 21579 sys_catalog.cc:455] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [sys.catalog]: SysCatalogTable state changed. Reason: New leader adac13120cf74bcd89709fd9fec9c67a. Latest consensus state: current_term: 1 leader_uuid: "adac13120cf74bcd89709fd9fec9c67a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adac13120cf74bcd89709fd9fec9c67a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40201 } } }
I20260925 19:04:26.529654 21579 sys_catalog.cc:458] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:26.529844 21589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:26.530668 21589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:26.531584 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:26.529305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:55833 (local address 127.0.0.1:40201)
0925 19:04:26.530126 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:26.530134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:26.530159 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:04:26.530300 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:26.530306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:26.530350 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:26.530512 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:26.531165 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:26.531168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:26.531169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:26.531172 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:26.531209 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:26.531225 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:26.531311 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:26.531386 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 19:04:26.531432 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":64,"threads_started":1}
I20260925 19:04:26.532358 21589 catalog_manager.cc:1347] Generated new cluster ID: 04120acba59146fb82e20f285bb8dac8
I20260925 19:04:26.532449 21589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:26.532508 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:26.543454 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:26.534554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:47065 (local address 127.0.0.1:40201)
0925 19:04:26.534589 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:26.534591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:26.542414 (+ 7823us) server_negotiation.cc:407] Connection header received
0925 19:04:26.542485 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:26.542486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:26.542490 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:26.542583 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:26.543208 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:26.543209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:26.543210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:26.543215 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:26.543230 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:26.543234 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:26.543289 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:26.543356 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 19:04:26.543370 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:04:26.543968 21589 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:26.544104 21589 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:26.552829 21589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adac13120cf74bcd89709fd9fec9c67a: Generated new TSK 0
I20260925 19:04:26.553005 21589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:26.558555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 21506
2026-09-25T19:04:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1284 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:26 dist-test-slave-49bz krb5kdc[21605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:26 dist-test-slave-49bz krb5kdc[21605](info): set up 2 sockets
Sep 25 19:04:26 dist-test-slave-49bz krb5kdc[21605](info): commencing operation
krb5kdc: starting...
W20260925 19:04:27.590109 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:27 dist-test-slave-49bz krb5kdc[21605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:27.709025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:27.787271 21618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:27.787510 21618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:27.787549 21618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:27.788662 21618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:27.788734 21618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:27.788758 21618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:27.788798 21618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:27.788821 21618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:27.790331 21618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:27.790603 21618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:27.790817 21618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:27.795320 21623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:27.795388 21624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:27.795421 21626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:27.795653 21618 server_base.cc:1034] running on GCE node
I20260925 19:04:27.795979 21618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:27.796257 21618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:27.797400 21618 hybrid_clock.cc:648] HybridClock initialized: now 1790363067797399 us; error 40 us; skew 500 ppm
Sep 25 19:04:27 dist-test-slave-49bz krb5kdc[21605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:27.803768 21618 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:27.804248 21618 webserver.cc:460] Webserver started at http://127.0.0.1:39527/ using document root <none> and password file <none>
I20260925 19:04:27.804473 21618 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:27.804543 21618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:27.804674 21618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:27.805686 21618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "a84c363df3234b21b10448093938400c"
format_stamp: "Formatted at 2026-09-25 19:04:27 on dist-test-slave-49bz"
server_key: "d0f28a51da8d1c35e7047f1759ceffd4"
server_key_iv: "84fa84ae9caecb5af671a0e1322cabcd"
server_key_version: "encryptionkey@0"
I20260925 19:04:27.806051 21618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "a84c363df3234b21b10448093938400c"
format_stamp: "Formatted at 2026-09-25 19:04:27 on dist-test-slave-49bz"
server_key: "d0f28a51da8d1c35e7047f1759ceffd4"
server_key_iv: "84fa84ae9caecb5af671a0e1322cabcd"
server_key_version: "encryptionkey@0"
I20260925 19:04:27.807374 21618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 19:04:27.808080 21633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:27.808267 21618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:27.808352 21618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "a84c363df3234b21b10448093938400c"
format_stamp: "Formatted at 2026-09-25 19:04:27 on dist-test-slave-49bz"
server_key: "d0f28a51da8d1c35e7047f1759ceffd4"
server_key_iv: "84fa84ae9caecb5af671a0e1322cabcd"
server_key_version: "encryptionkey@0"
I20260925 19:04:27.808404 21618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:27.828683 21618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:27.831103 21618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:27.831214 21618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:27.835001 21618 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34253
I20260925 19:04:27.835069 21685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34253 every 8 connection(s)
I20260925 19:04:27.835564 21618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:27.836310 21686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:27.838485 21686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c: Bootstrap starting.
I20260925 19:04:27.839064 21686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:27.839346 21686 log.cc:824] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:27.840037 21686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c: No bootstrap required, opened a new log
I20260925 19:04:27.841547 21686 raft_consensus.cc:348] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84c363df3234b21b10448093938400c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260925 19:04:27.841699 21686 raft_consensus.cc:374] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:27.841746 21686 raft_consensus.cc:729] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84c363df3234b21b10448093938400c, State: Initialized, Role: FOLLOWER
I20260925 19:04:27.841933 21686 consensus_queue.cc:260] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84c363df3234b21b10448093938400c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260925 19:04:27.842032 21686 raft_consensus.cc:388] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:27.842074 21686 raft_consensus.cc:482] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:27.842125 21686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:27.843037 21686 raft_consensus.cc:504] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84c363df3234b21b10448093938400c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260925 19:04:27.843154 21686 leader_election.cc:302] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a84c363df3234b21b10448093938400c; no voters:
I20260925 19:04:27.843339 21686 leader_election.cc:288] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:27.843389 21689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:27.843603 21689 raft_consensus.cc:686] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [term 1 LEADER]: Becoming Leader. State: Replica: a84c363df3234b21b10448093938400c, State: Running, Role: LEADER
I20260925 19:04:27.843609 21686 sys_catalog.cc:564] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:27.843701 21689 consensus_queue.cc:237] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84c363df3234b21b10448093938400c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260925 19:04:27.843910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 21618
I20260925 19:04:27.844080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:27.844278 392 external_mini_cluster.cc:1428] Setting key fad8a07bf0a7361fcd2e553d73e4d5fe
I20260925 19:04:27.845064 21690 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a84c363df3234b21b10448093938400c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84c363df3234b21b10448093938400c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } } }
I20260925 19:04:27.845064 21691 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84c363df3234b21b10448093938400c. Latest consensus state: current_term: 1 leader_uuid: "a84c363df3234b21b10448093938400c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84c363df3234b21b10448093938400c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } } }
I20260925 19:04:27.845180 21690 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:27.845185 21691 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c [sys.catalog]: This master's current role is: LEADER
W20260925 19:04:27.845724 21706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:04:27.845824 21706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:04:27.845924 21707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:27.846124 21707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:27.847766 21707 catalog_manager.cc:1347] Generated new cluster ID: 7dfffd8460284d6e90b5d8ef1fcfd2c2
I20260925 19:04:27.847822 21707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:04:27 dist-test-slave-49bz krb5kdc[21605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 19:04:27.855877 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:27.844715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:55309 (local address 127.0.0.1:34253)
0925 19:04:27.845285 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:27.845295 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:27.845330 (+ 35us) server_negotiation.cc:407] Connection header received
0925 19:04:27.845474 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:27.845504 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:27.845604 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:27.845767 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:27.846574 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:27.847752 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:27.848823 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:27.849398 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:27.849779 (+ 381us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:27.852208 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:27.852211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:27.852212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:27.852223 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:27.854078 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:27.854507 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:27.854509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:27.854509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:27.854600 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:27.854788 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:27.854789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:27.854789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:27.855042 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:27.855474 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:27.855568 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 19:04:27.855661 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":88,"threads_started":1}
I20260925 19:04:27.856896 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:27.857380 21707 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:27.858026 21707 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:27.862653 21707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a84c363df3234b21b10448093938400c: Generated new TSK 0
I20260925 19:04:27.862808 21707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:27.870158 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:27.858540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:46273 (local address 127.0.0.1:34253)
0925 19:04:27.858576 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:27.858578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:27.858690 (+ 112us) server_negotiation.cc:407] Connection header received
0925 19:04:27.858756 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:27.858757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:27.858766 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:27.858818 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:27.859296 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:27.860727 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:27.864792 (+ 4065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:27.865146 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:27.865220 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:27.866813 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:27.866814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:27.866814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:27.866821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:27.868734 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:27.869233 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:27.869234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:27.869235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:27.869292 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:27.869570 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:27.869570 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:27.869571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:27.869694 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:27.869765 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:27.870012 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 19:04:27.870080 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:04:27.872524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 21618
W20260925 19:04:27.877575 21603 connection.cc:535] client connection to 127.0.0.1:34253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34253): Connection reset by peer (error 104)
2026-09-25T19:04:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:27 dist-test-slave-49bz krb5kdc[21717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:27 dist-test-slave-49bz krb5kdc[21717](info): set up 2 sockets
Sep 25 19:04:27 dist-test-slave-49bz krb5kdc[21717](info): commencing operation
krb5kdc: starting...
W20260925 19:04:28.926427 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:29 dist-test-slave-49bz krb5kdc[21717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:29.041543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:29.119167 21730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:29.119376 21730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:29.119400 21730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:29.120612 21730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:29.120656 21730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:29.120671 21730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:29.120683 21730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:29.120695 21730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:29.122175 21730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:29.122465 21730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:29.122697 21730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:29.127295 21735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:29.127322 21736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:29.127439 21730 server_base.cc:1034] running on GCE node
W20260925 19:04:29.127303 21738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:29.127879 21730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:29.128165 21730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:29.129307 21730 hybrid_clock.cc:648] HybridClock initialized: now 1790363069129297 us; error 26 us; skew 500 ppm
Sep 25 19:04:29 dist-test-slave-49bz krb5kdc[21717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:29.131803 21730 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:29.132265 21730 webserver.cc:460] Webserver started at http://127.0.0.1:44615/ using document root <none> and password file <none>
I20260925 19:04:29.132507 21730 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:29.132579 21730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:29.132731 21730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:29.133772 21730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "93d30b8e3b97413f919398f1edd63c19"
format_stamp: "Formatted at 2026-09-25 19:04:29 on dist-test-slave-49bz"
server_key: "df87d843389e48ae9cb36418ddeb6b47"
server_key_iv: "fa4cd9070af0815b5604378f4c045736"
server_key_version: "encryptionkey@0"
I20260925 19:04:29.134125 21730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "93d30b8e3b97413f919398f1edd63c19"
format_stamp: "Formatted at 2026-09-25 19:04:29 on dist-test-slave-49bz"
server_key: "df87d843389e48ae9cb36418ddeb6b47"
server_key_iv: "fa4cd9070af0815b5604378f4c045736"
server_key_version: "encryptionkey@0"
I20260925 19:04:29.135460 21730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:29.136230 21745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:29.136422 21730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:29.136533 21730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "93d30b8e3b97413f919398f1edd63c19"
format_stamp: "Formatted at 2026-09-25 19:04:29 on dist-test-slave-49bz"
server_key: "df87d843389e48ae9cb36418ddeb6b47"
server_key_iv: "fa4cd9070af0815b5604378f4c045736"
server_key_version: "encryptionkey@0"
I20260925 19:04:29.136610 21730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:29.160482 21730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:29.162918 21730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:29.163086 21730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:29.167146 21730 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39189
I20260925 19:04:29.167183 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39189 every 8 connection(s)
I20260925 19:04:29.167527 21730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:29.168460 21798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:29.170797 21798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19: Bootstrap starting.
I20260925 19:04:29.171394 21798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:29.171691 21798 log.cc:824] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:29.172406 21798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19: No bootstrap required, opened a new log
I20260925 19:04:29.173969 21798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d30b8e3b97413f919398f1edd63c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39189 } }
I20260925 19:04:29.174103 21798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:29.174126 21798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d30b8e3b97413f919398f1edd63c19, State: Initialized, Role: FOLLOWER
I20260925 19:04:29.174350 21798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d30b8e3b97413f919398f1edd63c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39189 } }
I20260925 19:04:29.174419 21798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:29.174441 21798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:29.174492 21798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:29.174993 21798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d30b8e3b97413f919398f1edd63c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39189 } }
I20260925 19:04:29.175076 21798 leader_election.cc:302] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93d30b8e3b97413f919398f1edd63c19; no voters:
I20260925 19:04:29.175288 21798 leader_election.cc:288] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:29.175359 21801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:29.175534 21801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [term 1 LEADER]: Becoming Leader. State: Replica: 93d30b8e3b97413f919398f1edd63c19, State: Running, Role: LEADER
I20260925 19:04:29.175539 21798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:29.175657 21801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d30b8e3b97413f919398f1edd63c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39189 } }
I20260925 19:04:29.176077 21803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d30b8e3b97413f919398f1edd63c19. Latest consensus state: current_term: 1 leader_uuid: "93d30b8e3b97413f919398f1edd63c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d30b8e3b97413f919398f1edd63c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39189 } } }
I20260925 19:04:29.176165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 21730
I20260925 19:04:29.176208 21802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93d30b8e3b97413f919398f1edd63c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d30b8e3b97413f919398f1edd63c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39189 } } }
I20260925 19:04:29.176215 21803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:29.176272 21802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:29.176291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:29.176445 21810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:29.176465 392 external_mini_cluster.cc:1428] Setting key f5adf26912b46284b6994e32f7c1416d
I20260925 19:04:29.177160 21810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:29.178809 21810 catalog_manager.cc:1347] Generated new cluster ID: 9186b3ed8c274d1cbddcd1b7c003cce1
I20260925 19:04:29.178865 21810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:29.183095 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:29.177099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:58343 (local address 127.0.0.1:39189)
0925 19:04:29.177820 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:29.177828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:29.177860 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:04:29.177977 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:29.177983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:29.178036 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:29.178188 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:29.178447 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:29.179725 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:29.180894 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:29.181492 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:29.181592 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:29.182401 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:29.182405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:29.182405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:29.182605 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:29.182648 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:29.182663 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:29.182738 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:29.182873 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 19:04:29.182898 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":103,"threads_started":1}
I20260925 19:04:29.184105 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:29.189105 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:29.185646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:46273 (local address 127.0.0.1:39189)
0925 19:04:29.185681 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:29.185683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:29.185821 (+ 138us) server_negotiation.cc:407] Connection header received
0925 19:04:29.185868 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:29.185869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:29.185880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:29.185931 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:29.186071 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:29.186808 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:29.187813 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:29.188101 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:29.188160 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:29.188855 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:29.188857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:29.188857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:29.188862 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:29.188876 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:29.188881 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:29.188938 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:29.189022 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 19:04:29.189033 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:04:29.190378 21810 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:29.190977 21810 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:29.197745 21810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93d30b8e3b97413f919398f1edd63c19: Generated new TSK 0
I20260925 19:04:29.197922 21810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:29.203787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 21730
W20260925 19:04:29.207937 21715 connection.cc:535] client connection to 127.0.0.1:39189 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39189): Connection reset by peer (error 104)
2026-09-25T19:04:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:29 dist-test-slave-49bz krb5kdc[21829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:29 dist-test-slave-49bz krb5kdc[21829](info): set up 2 sockets
Sep 25 19:04:29 dist-test-slave-49bz krb5kdc[21829](info): commencing operation
krb5kdc: starting...
W20260925 19:04:30.214432 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:30 dist-test-slave-49bz krb5kdc[21829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 19:04:30.324895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:30.404150 21842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:30.404376 21842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:30.404413 21842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:30.405551 21842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:30.405618 21842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:30.405644 21842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:30.405675 21842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:30.405699 21842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:30.407207 21842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:30.407482 21842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:30.407703 21842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:04:30.413184 21842 server_base.cc:1034] running on GCE node
W20260925 19:04:30.413166 21850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:30.413166 21847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:30.413158 21848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:30.413519 21842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:30.413802 21842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:30.414939 21842 hybrid_clock.cc:648] HybridClock initialized: now 1790363070414922 us; error 35 us; skew 500 ppm
Sep 25 19:04:30 dist-test-slave-49bz krb5kdc[21829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:30.417204 21842 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 19:04:30.417661 21842 webserver.cc:460] Webserver started at http://127.0.0.1:33277/ using document root <none> and password file <none>
I20260925 19:04:30.417896 21842 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:30.417964 21842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:30.418100 21842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:30.419163 21842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "578970acea5d42a8a3435515e932c206"
format_stamp: "Formatted at 2026-09-25 19:04:30 on dist-test-slave-49bz"
server_key: "cdd06338d63fc11297b8fa44cd948efc"
server_key_iv: "af33e540f43269365b7eba0418212bc9"
server_key_version: "encryptionkey@0"
I20260925 19:04:30.419521 21842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "578970acea5d42a8a3435515e932c206"
format_stamp: "Formatted at 2026-09-25 19:04:30 on dist-test-slave-49bz"
server_key: "cdd06338d63fc11297b8fa44cd948efc"
server_key_iv: "af33e540f43269365b7eba0418212bc9"
server_key_version: "encryptionkey@0"
I20260925 19:04:30.420776 21842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:30.421452 21857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:30.421612 21842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:30.421697 21842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "578970acea5d42a8a3435515e932c206"
format_stamp: "Formatted at 2026-09-25 19:04:30 on dist-test-slave-49bz"
server_key: "cdd06338d63fc11297b8fa44cd948efc"
server_key_iv: "af33e540f43269365b7eba0418212bc9"
server_key_version: "encryptionkey@0"
I20260925 19:04:30.421747 21842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:30.424916 21842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:30.426697 21842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:30.426795 21842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:30.430508 21842 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45917
I20260925 19:04:30.430537 21909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45917 every 8 connection(s)
I20260925 19:04:30.430859 21842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:30.431784 21910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:30.433859 21910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206: Bootstrap starting.
I20260925 19:04:30.434423 21910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:30.434702 21910 log.cc:824] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:30.435328 21910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206: No bootstrap required, opened a new log
I20260925 19:04:30.436832 21910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578970acea5d42a8a3435515e932c206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260925 19:04:30.436980 21910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:30.437027 21910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 578970acea5d42a8a3435515e932c206, State: Initialized, Role: FOLLOWER
I20260925 19:04:30.437161 21910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578970acea5d42a8a3435515e932c206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260925 19:04:30.437239 21910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:30.437280 21910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:30.437330 21910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:30.437829 21910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578970acea5d42a8a3435515e932c206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260925 19:04:30.437938 21910 leader_election.cc:302] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 578970acea5d42a8a3435515e932c206; no voters:
I20260925 19:04:30.438108 21910 leader_election.cc:288] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:30.438167 21913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:30.438373 21910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:30.438380 21913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [term 1 LEADER]: Becoming Leader. State: Replica: 578970acea5d42a8a3435515e932c206, State: Running, Role: LEADER
I20260925 19:04:30.438484 21913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578970acea5d42a8a3435515e932c206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } }
I20260925 19:04:30.438968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 21842
I20260925 19:04:30.439088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:30.439286 392 external_mini_cluster.cc:1428] Setting key e7fa4912fc15eb38bd92d06ee7bea4d6
I20260925 19:04:30.439764 21914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "578970acea5d42a8a3435515e932c206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578970acea5d42a8a3435515e932c206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } } }
I20260925 19:04:30.439778 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 578970acea5d42a8a3435515e932c206. Latest consensus state: current_term: 1 leader_uuid: "578970acea5d42a8a3435515e932c206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578970acea5d42a8a3435515e932c206" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45917 } } }
I20260925 19:04:30.439968 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:30.440176 21914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:30.440847 21930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:30.441349 21930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:30.441596 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:30.439974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:50073 (local address 127.0.0.1:45917)
0925 19:04:30.440149 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:30.440158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:30.440189 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:04:30.440313 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:30.440320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:30.440352 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:30.440509 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:30.441127 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:30.441130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:30.441130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:30.441134 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:30.441195 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:30.441213 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:30.441293 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:30.441375 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 19:04:30.441417 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260925 19:04:30.442637 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:30.443148 21930 catalog_manager.cc:1347] Generated new cluster ID: c53e21493b8441d89bdd601888b378cf
I20260925 19:04:30.443195 21930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:30.445247 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:30.444100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.18:52281 (local address 127.0.0.1:45917)
0925 19:04:30.444161 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:30.444162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:30.444348 (+ 186us) server_negotiation.cc:407] Connection header received
0925 19:04:30.444403 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:30.444404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:30.444408 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:30.444460 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:30.445009 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:30.445010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:30.445010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:30.445014 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:30.445028 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:30.445032 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:30.445087 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:30.445162 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 19:04:30.445173 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 19:04:30.448935 21930 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:30.449079 21930 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:30.465626 21930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 578970acea5d42a8a3435515e932c206: Generated new TSK 0
I20260925 19:04:30.465806 21930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:30.469983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 21842
2026-09-25T19:04:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:30 dist-test-slave-49bz krb5kdc[21941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:30 dist-test-slave-49bz krb5kdc[21941](info): set up 2 sockets
Sep 25 19:04:30 dist-test-slave-49bz krb5kdc[21941](info): commencing operation
krb5kdc: starting...
W20260925 19:04:31.478415 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:31 dist-test-slave-49bz krb5kdc[21941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:31.590682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43635
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:31.666141 21954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:31.666388 21954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:31.666427 21954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:31.667493 21954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:31.667563 21954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:31.667587 21954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:31.667608 21954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:31.667635 21954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:31.669005 21954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:31.669268 21954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:31.669484 21954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:31.674036 21959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:31.674072 21960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:31.674180 21954 server_base.cc:1034] running on GCE node
W20260925 19:04:31.674094 21962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:31.674556 21954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:31.674847 21954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:31.676013 21954 hybrid_clock.cc:648] HybridClock initialized: now 1790363071676002 us; error 27 us; skew 500 ppm
Sep 25 19:04:31 dist-test-slave-49bz krb5kdc[21941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:31.678365 21954 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:31.678824 21954 webserver.cc:460] Webserver started at http://127.0.98.62:36059/ using document root <none> and password file <none>
I20260925 19:04:31.679040 21954 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:31.679104 21954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:31.679247 21954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:31.680223 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "a2f8274354414c30ab1a3ec7940996a1"
format_stamp: "Formatted at 2026-09-25 19:04:31 on dist-test-slave-49bz"
server_key: "db0a351d837d3e14ff4e9daa5111bd75"
server_key_iv: "0905600818f31d316bfebab4bf55ed2b"
server_key_version: "encryptionkey@0"
I20260925 19:04:31.680568 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2f8274354414c30ab1a3ec7940996a1"
format_stamp: "Formatted at 2026-09-25 19:04:31 on dist-test-slave-49bz"
server_key: "db0a351d837d3e14ff4e9daa5111bd75"
server_key_iv: "0905600818f31d316bfebab4bf55ed2b"
server_key_version: "encryptionkey@0"
I20260925 19:04:31.681878 21954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:04:31.682670 21969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:31.682847 21954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:31.682943 21954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "a2f8274354414c30ab1a3ec7940996a1"
format_stamp: "Formatted at 2026-09-25 19:04:31 on dist-test-slave-49bz"
server_key: "db0a351d837d3e14ff4e9daa5111bd75"
server_key_iv: "0905600818f31d316bfebab4bf55ed2b"
server_key_version: "encryptionkey@0"
I20260925 19:04:31.683024 21954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:31.695766 21954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:31.697407 21954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:31.697563 21954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:31.701448 21954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43635
I20260925 19:04:31.701476 22021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43635 every 8 connection(s)
I20260925 19:04:31.701879 21954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:31.702739 22022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:31.704629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 21954
I20260925 19:04:31.704783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:31.704918 22022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1: Bootstrap starting.
I20260925 19:04:31.704953 392 external_mini_cluster.cc:1428] Setting key f1201f37a957143ed564b7807b3b975f
I20260925 19:04:31.705632 22022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:31.705945 22022 log.cc:824] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:31.706732 22022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1: No bootstrap required, opened a new log
I20260925 19:04:31.708920 22022 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f8274354414c30ab1a3ec7940996a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43635 } }
I20260925 19:04:31.709038 22022 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:31.709059 22022 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f8274354414c30ab1a3ec7940996a1, State: Initialized, Role: FOLLOWER
I20260925 19:04:31.709182 22022 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f8274354414c30ab1a3ec7940996a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43635 } }
I20260925 19:04:31.709250 22022 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:31.709270 22022 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:31.709344 22022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:31.709820 22022 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f8274354414c30ab1a3ec7940996a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43635 } }
I20260925 19:04:31.709933 22022 leader_election.cc:302] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2f8274354414c30ab1a3ec7940996a1; no voters:
I20260925 19:04:31.710083 22022 leader_election.cc:288] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:31.710114 22027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:31.710358 22022 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:31.710579 22027 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [term 1 LEADER]: Becoming Leader. State: Replica: a2f8274354414c30ab1a3ec7940996a1, State: Running, Role: LEADER
I20260925 19:04:31.710690 22027 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f8274354414c30ab1a3ec7940996a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43635 } }
I20260925 19:04:31.711066 22028 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2f8274354414c30ab1a3ec7940996a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f8274354414c30ab1a3ec7940996a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43635 } } }
I20260925 19:04:31.711150 22028 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:31.711277 22029 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2f8274354414c30ab1a3ec7940996a1. Latest consensus state: current_term: 1 leader_uuid: "a2f8274354414c30ab1a3ec7940996a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f8274354414c30ab1a3ec7940996a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43635 } } }
I20260925 19:04:31.711333 22036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:31.711338 22029 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:31.712049 22036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 19:04:31 dist-test-slave-49bz krb5kdc[21941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:31.714107 22036 catalog_manager.cc:1347] Generated new cluster ID: e5b31436ce3047209c9ed9f683391f96
I20260925 19:04:31.714160 22036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:31.716246 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:31.705464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55673 (local address 127.0.98.62:43635)
0925 19:04:31.705669 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:31.705678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:31.705710 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:04:31.706358 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:31.706365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:31.706431 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:31.706571 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:31.707202 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:31.708588 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:31.709454 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:31.709861 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:31.709987 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:31.712892 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:31.712896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:31.712896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:31.712905 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:31.714730 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:31.715121 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:31.715124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:31.715124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:31.715210 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:31.715445 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:31.715446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:31.715446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:31.715682 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:31.715864 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:31.715947 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 19:04:31.716041 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260925 19:04:31.717118 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:31.724280 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:31.718422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44637 (local address 127.0.98.62:43635)
0925 19:04:31.718475 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:31.718477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:31.718516 (+ 39us) server_negotiation.cc:407] Connection header received
0925 19:04:31.718579 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:31.718580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:31.718587 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:31.718627 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:31.719116 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:31.719575 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:31.720241 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:31.720456 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:31.720526 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:31.721631 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:31.721632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:31.721632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:31.721639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:31.723128 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:31.723493 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:31.723495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:31.723495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:31.723551 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:31.723784 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:31.723785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:31.723786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:31.723890 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:31.724010 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:31.724131 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 19:04:31.724191 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:04:31.753314 22036 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:31.754031 22036 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:31.764894 22036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2f8274354414c30ab1a3ec7940996a1: Generated new TSK 0
I20260925 19:04:31.765079 22036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:31.771849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 21954
W20260925 19:04:31.776407 21939 connection.cc:535] client connection to 127.0.98.62:43635 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43635): Connection reset by peer (error 104)
2026-09-25T19:04:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:31 dist-test-slave-49bz krb5kdc[22053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:31 dist-test-slave-49bz krb5kdc[22053](info): set up 2 sockets
Sep 25 19:04:31 dist-test-slave-49bz krb5kdc[22053](info): commencing operation
krb5kdc: starting...
W20260925 19:04:32.794134 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:32 dist-test-slave-49bz krb5kdc[22053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:32.906230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38935
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:32.985689 22066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:32.985944 22066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:32.985983 22066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:32.987218 22066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:32.987288 22066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:32.987313 22066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:32.987342 22066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:32.987372 22066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:32.988806 22066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:32.989086 22066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:32.989306 22066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:32.993943 22074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:32.994074 22066 server_base.cc:1034] running on GCE node
W20260925 19:04:32.994006 22072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:32.993942 22071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:32.994398 22066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:32.994709 22066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:32.995875 22066 hybrid_clock.cc:648] HybridClock initialized: now 1790363072995870 us; error 35 us; skew 500 ppm
Sep 25 19:04:32 dist-test-slave-49bz krb5kdc[22053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:33.002033 22066 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:33.002544 22066 webserver.cc:460] Webserver started at http://127.0.98.62:39107/ using document root <none> and password file <none>
I20260925 19:04:33.002774 22066 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:33.002842 22066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:33.002995 22066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:33.004069 22066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "466db4202f9b48acb2a9518f41ab4cc4"
format_stamp: "Formatted at 2026-09-25 19:04:32 on dist-test-slave-49bz"
server_key: "14c09ccee834577d04920a22d0b6f388"
server_key_iv: "ca5c1f72846c3ce87ddfd1004add563d"
server_key_version: "encryptionkey@0"
I20260925 19:04:33.004436 22066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "466db4202f9b48acb2a9518f41ab4cc4"
format_stamp: "Formatted at 2026-09-25 19:04:32 on dist-test-slave-49bz"
server_key: "14c09ccee834577d04920a22d0b6f388"
server_key_iv: "ca5c1f72846c3ce87ddfd1004add563d"
server_key_version: "encryptionkey@0"
I20260925 19:04:33.005752 22066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:33.006508 22081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:33.006685 22066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:33.006788 22066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "466db4202f9b48acb2a9518f41ab4cc4"
format_stamp: "Formatted at 2026-09-25 19:04:32 on dist-test-slave-49bz"
server_key: "14c09ccee834577d04920a22d0b6f388"
server_key_iv: "ca5c1f72846c3ce87ddfd1004add563d"
server_key_version: "encryptionkey@0"
I20260925 19:04:33.006860 22066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:33.018771 22066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:33.020673 22066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:33.020831 22066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:33.024843 22066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38935
I20260925 19:04:33.024878 22133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38935 every 8 connection(s)
I20260925 19:04:33.025221 22066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:33.026036 22134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:33.028311 22134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4: Bootstrap starting.
I20260925 19:04:33.028844 22134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:33.029072 22134 log.cc:824] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:33.029754 22134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4: No bootstrap required, opened a new log
I20260925 19:04:33.030860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 22066
I20260925 19:04:33.030994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:33.031168 392 external_mini_cluster.cc:1428] Setting key 3eeab6e4c21e7d572eb82008fa9cd9a2
I20260925 19:04:33.031533 22134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466db4202f9b48acb2a9518f41ab4cc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38935 } }
I20260925 19:04:33.031749 22134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:33.031793 22134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 466db4202f9b48acb2a9518f41ab4cc4, State: Initialized, Role: FOLLOWER
I20260925 19:04:33.031994 22134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466db4202f9b48acb2a9518f41ab4cc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38935 } }
I20260925 19:04:33.032063 22134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:33.032084 22134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:33.032156 22134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:33.033143 22134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466db4202f9b48acb2a9518f41ab4cc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38935 } }
I20260925 19:04:33.033229 22134 leader_election.cc:302] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 466db4202f9b48acb2a9518f41ab4cc4; no voters:
I20260925 19:04:33.033428 22134 leader_election.cc:288] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:33.033448 22139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:33.033552 22139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 466db4202f9b48acb2a9518f41ab4cc4, State: Running, Role: LEADER
I20260925 19:04:33.033685 22134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:33.033663 22139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466db4202f9b48acb2a9518f41ab4cc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38935 } }
I20260925 19:04:33.034102 22140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "466db4202f9b48acb2a9518f41ab4cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466db4202f9b48acb2a9518f41ab4cc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38935 } } }
I20260925 19:04:33.034112 22141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 466db4202f9b48acb2a9518f41ab4cc4. Latest consensus state: current_term: 1 leader_uuid: "466db4202f9b48acb2a9518f41ab4cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466db4202f9b48acb2a9518f41ab4cc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38935 } } }
I20260925 19:04:33.034296 22141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:33.034474 22140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:33.034629 22146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:33.035377 22146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:33.036940 22146 catalog_manager.cc:1347] Generated new cluster ID: a40f749d2651421495e98510d53b1f5d
I20260925 19:04:33.037029 22146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:33.037243 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:33.031702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60907 (local address 127.0.98.62:38935)
0925 19:04:33.031919 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:33.031927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:33.031952 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:04:33.032098 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:33.032105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:33.032178 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:33.032313 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:33.032572 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:33.033647 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:33.034781 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:33.035475 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:33.035601 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:33.036483 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:33.036487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:33.036487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:33.036723 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:33.036760 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:33.036774 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:33.036853 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:33.037041 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 19:04:33.037070 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260925 19:04:33.038123 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:33.045120 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:33.039476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59395 (local address 127.0.98.62:38935)
0925 19:04:33.039510 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:33.039512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:33.042434 (+ 2922us) server_negotiation.cc:407] Connection header received
0925 19:04:33.042506 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:33.042507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:33.042518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:33.042574 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:33.042719 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:33.043187 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:33.043899 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:33.044128 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:33.044200 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:33.044853 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:33.044854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:33.044854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:33.044858 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:33.044871 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:33.044875 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:33.044932 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:33.045004 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 19:04:33.045016 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:04:33.055508 22146 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:33.056020 22146 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:33.059655 22146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 466db4202f9b48acb2a9518f41ab4cc4: Generated new TSK 0
I20260925 19:04:33.059845 22146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:33.068300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 22066
W20260925 19:04:33.072721 22051 connection.cc:535] client connection to 127.0.98.62:38935 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38935): Connection reset by peer (error 104)
2026-09-25T19:04:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:33 dist-test-slave-49bz krb5kdc[22165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:33 dist-test-slave-49bz krb5kdc[22165](info): set up 2 sockets
Sep 25 19:04:33 dist-test-slave-49bz krb5kdc[22165](info): commencing operation
krb5kdc: starting...
W20260925 19:04:34.078392 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:34 dist-test-slave-49bz krb5kdc[22165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:34.192685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33403
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:34.273833 22178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:34.274048 22178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:34.274072 22178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:34.275266 22178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:34.275312 22178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:34.275327 22178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:34.275341 22178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:34.275352 22178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:34.276759 22178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:34.277011 22178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:34.277191 22178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:34.281931 22186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:34.281898 22183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:34.282152 22178 server_base.cc:1034] running on GCE node
W20260925 19:04:34.281895 22184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:34.282522 22178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:34.282859 22178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:34.284018 22178 hybrid_clock.cc:648] HybridClock initialized: now 1790363074284008 us; error 31 us; skew 500 ppm
Sep 25 19:04:34 dist-test-slave-49bz krb5kdc[22165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:34.286623 22178 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:34.287164 22178 webserver.cc:460] Webserver started at http://127.0.98.62:40749/ using document root <none> and password file <none>
I20260925 19:04:34.287401 22178 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:34.287453 22178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:34.287600 22178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:34.288683 22178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "f237959cd8f7455ea871bdc515087dfc"
format_stamp: "Formatted at 2026-09-25 19:04:34 on dist-test-slave-49bz"
server_key: "508104e49a23978f8c7b1eb37f4c08fe"
server_key_iv: "a12c306f6db5f4924f979b507286bdbc"
server_key_version: "encryptionkey@0"
I20260925 19:04:34.289047 22178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "f237959cd8f7455ea871bdc515087dfc"
format_stamp: "Formatted at 2026-09-25 19:04:34 on dist-test-slave-49bz"
server_key: "508104e49a23978f8c7b1eb37f4c08fe"
server_key_iv: "a12c306f6db5f4924f979b507286bdbc"
server_key_version: "encryptionkey@0"
I20260925 19:04:34.290402 22178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 19:04:34.291196 22193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:34.291384 22178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:34.291508 22178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "f237959cd8f7455ea871bdc515087dfc"
format_stamp: "Formatted at 2026-09-25 19:04:34 on dist-test-slave-49bz"
server_key: "508104e49a23978f8c7b1eb37f4c08fe"
server_key_iv: "a12c306f6db5f4924f979b507286bdbc"
server_key_version: "encryptionkey@0"
I20260925 19:04:34.291581 22178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:34.294981 22178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:34.296839 22178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:34.296986 22178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:34.300940 22178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33403
I20260925 19:04:34.300971 22245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33403 every 8 connection(s)
I20260925 19:04:34.301299 22178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:34.302243 22246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:34.304507 22246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc: Bootstrap starting.
I20260925 19:04:34.305081 22246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:34.305356 22246 log.cc:824] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:34.306058 22246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc: No bootstrap required, opened a new log
I20260925 19:04:34.307420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 22178
I20260925 19:04:34.307562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:34.307744 392 external_mini_cluster.cc:1428] Setting key 7aab2eceb009bda5a6513499556622d4
I20260925 19:04:34.308100 22246 raft_consensus.cc:348] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237959cd8f7455ea871bdc515087dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33403 } }
I20260925 19:04:34.308286 22246 raft_consensus.cc:374] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:34.308350 22246 raft_consensus.cc:729] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f237959cd8f7455ea871bdc515087dfc, State: Initialized, Role: FOLLOWER
I20260925 19:04:34.308532 22246 consensus_queue.cc:260] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237959cd8f7455ea871bdc515087dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33403 } }
I20260925 19:04:34.308617 22246 raft_consensus.cc:388] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:34.308648 22246 raft_consensus.cc:482] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:34.308686 22246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:34.309465 22246 raft_consensus.cc:504] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237959cd8f7455ea871bdc515087dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33403 } }
I20260925 19:04:34.309580 22246 leader_election.cc:302] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f237959cd8f7455ea871bdc515087dfc; no voters:
I20260925 19:04:34.309738 22246 leader_election.cc:288] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:34.309782 22251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:34.309983 22251 raft_consensus.cc:686] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [term 1 LEADER]: Becoming Leader. State: Replica: f237959cd8f7455ea871bdc515087dfc, State: Running, Role: LEADER
I20260925 19:04:34.310012 22246 sys_catalog.cc:564] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:34.310595 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:34.308266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37545 (local address 127.0.98.62:33403)
0925 19:04:34.308446 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:34.308455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:34.308487 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:04:34.308609 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:34.308616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:34.308648 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:34.308789 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:34.310090 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:34.310093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:34.310094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:34.310098 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:34.310135 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:34.310149 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:34.310213 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:34.310358 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 19:04:34.310400 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260925 19:04:34.310815 22251 consensus_queue.cc:237] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237959cd8f7455ea871bdc515087dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33403 } }
I20260925 19:04:34.311493 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:34.311434 22253 sys_catalog.cc:455] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f237959cd8f7455ea871bdc515087dfc. Latest consensus state: current_term: 1 leader_uuid: "f237959cd8f7455ea871bdc515087dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237959cd8f7455ea871bdc515087dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33403 } } }
I20260925 19:04:34.311764 22253 sys_catalog.cc:458] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:34.311434 22252 sys_catalog.cc:455] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f237959cd8f7455ea871bdc515087dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f237959cd8f7455ea871bdc515087dfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33403 } } }
I20260925 19:04:34.311933 22252 sys_catalog.cc:458] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:34.312116 22266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:34.312641 22266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:34.314445 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:34.313041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51437 (local address 127.0.98.62:33403)
0925 19:04:34.313073 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:34.313075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:34.313475 (+ 400us) server_negotiation.cc:407] Connection header received
0925 19:04:34.313545 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:34.313546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:34.313562 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:34.313605 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:34.314206 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:34.314207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:34.314208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:34.314212 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:34.314226 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:34.314230 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:34.314323 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:34.314376 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 19:04:34.314389 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:04:34.314631 22266 catalog_manager.cc:1347] Generated new cluster ID: f1f0d22afd1949f293c67ad08ac420e6
I20260925 19:04:34.314678 22266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:34.333838 22266 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:34.334035 22266 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:34.347537 22266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f237959cd8f7455ea871bdc515087dfc: Generated new TSK 0
I20260925 19:04:34.347757 22266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:34.354107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 22178
2026-09-25T19:04:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:34 dist-test-slave-49bz krb5kdc[22277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:34 dist-test-slave-49bz krb5kdc[22277](info): set up 2 sockets
Sep 25 19:04:34 dist-test-slave-49bz krb5kdc[22277](info): commencing operation
krb5kdc: starting...
W20260925 19:04:35.362320 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:35 dist-test-slave-49bz krb5kdc[22277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:35.474823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38571
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:35.554706 22290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:35.554935 22290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:35.554975 22290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:35.556125 22290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:35.556188 22290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:35.556212 22290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:35.556233 22290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:35.556259 22290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:35.557725 22290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:35.557988 22290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:35.558207 22290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:35.562884 22298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:35.562855 22296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:35.562871 22295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:35.562944 22290 server_base.cc:1034] running on GCE node
I20260925 19:04:35.563320 22290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:35.563622 22290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:35.564790 22290 hybrid_clock.cc:648] HybridClock initialized: now 1790363075564782 us; error 35 us; skew 500 ppm
Sep 25 19:04:35 dist-test-slave-49bz krb5kdc[22277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:35.567145 22290 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:35.567553 22290 webserver.cc:460] Webserver started at http://127.0.98.62:44723/ using document root <none> and password file <none>
I20260925 19:04:35.567767 22290 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:35.567832 22290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:35.567994 22290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:35.569075 22290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9"
format_stamp: "Formatted at 2026-09-25 19:04:35 on dist-test-slave-49bz"
server_key: "bdc31a8e6a53d02eea13574547059aa3"
server_key_iv: "a9c0206fd2a2dae1a9f96ca16379aa21"
server_key_version: "encryptionkey@0"
I20260925 19:04:35.569438 22290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9"
format_stamp: "Formatted at 2026-09-25 19:04:35 on dist-test-slave-49bz"
server_key: "bdc31a8e6a53d02eea13574547059aa3"
server_key_iv: "a9c0206fd2a2dae1a9f96ca16379aa21"
server_key_version: "encryptionkey@0"
I20260925 19:04:35.570904 22290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:35.571640 22305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:35.571854 22290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:35.571943 22290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9"
format_stamp: "Formatted at 2026-09-25 19:04:35 on dist-test-slave-49bz"
server_key: "bdc31a8e6a53d02eea13574547059aa3"
server_key_iv: "a9c0206fd2a2dae1a9f96ca16379aa21"
server_key_version: "encryptionkey@0"
I20260925 19:04:35.571995 22290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:35.589865 22290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:35.591845 22290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:35.591954 22290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:35.595883 22290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38571
I20260925 19:04:35.595989 22357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38571 every 8 connection(s)
I20260925 19:04:35.596271 22290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:35.597198 22358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:35.599165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 22290
I20260925 19:04:35.599285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:35.599479 392 external_mini_cluster.cc:1428] Setting key 97e930a44079fa04c0397d6f6d2fb089
I20260925 19:04:35.599859 22358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9: Bootstrap starting.
I20260925 19:04:35.600515 22358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:35.600806 22358 log.cc:824] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:35.601492 22358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9: No bootstrap required, opened a new log
I20260925 19:04:35.603312 22358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } }
I20260925 19:04:35.603475 22358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:35.603523 22358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb7b13a9314c1ca942fdbd4e1d6db9, State: Initialized, Role: FOLLOWER
I20260925 19:04:35.603691 22358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } }
I20260925 19:04:35.603783 22358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:35.603848 22358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:35.603899 22358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:35.604589 22358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } }
I20260925 19:04:35.604710 22358 leader_election.cc:302] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ecb7b13a9314c1ca942fdbd4e1d6db9; no voters:
I20260925 19:04:35.604923 22358 leader_election.cc:288] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:35.604950 22363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:35.605077 22363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb7b13a9314c1ca942fdbd4e1d6db9, State: Running, Role: LEADER
I20260925 19:04:35.605186 22358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:35.605209 22363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } }
I20260925 19:04:35.605563 22363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ecb7b13a9314c1ca942fdbd4e1d6db9. Latest consensus state: current_term: 1 leader_uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } } }
I20260925 19:04:35.605579 22365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecb7b13a9314c1ca942fdbd4e1d6db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } } }
I20260925 19:04:35.605692 22363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:35.605921 22365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:35.606103 22371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 19:04:35 dist-test-slave-49bz krb5kdc[22277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:35.606958 22371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:35.608544 22371 catalog_manager.cc:1347] Generated new cluster ID: a44d67fc5097481bae67375d38a9700f
I20260925 19:04:35.608596 22371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:35.610466 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:35.599995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36565 (local address 127.0.98.62:38571)
0925 19:04:35.600187 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:35.600195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:35.600219 (+ 24us) server_negotiation.cc:407] Connection header received
0925 19:04:35.600681 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:35.600689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:35.600735 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:35.600880 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:35.601528 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:35.602502 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:35.603254 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:35.603736 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:35.603852 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:35.606879 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:35.606883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:35.606883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:35.606895 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:35.608838 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:35.609226 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:35.609228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:35.609229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:35.609345 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:35.609561 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:35.609562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:35.609562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:35.609844 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:35.609972 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:35.610104 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 19:04:35.610232 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260925 19:04:35.611380 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:35.620046 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:35.612746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57755 (local address 127.0.98.62:38571)
0925 19:04:35.612777 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:35.612779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:35.612900 (+ 121us) server_negotiation.cc:407] Connection header received
0925 19:04:35.612955 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:35.612956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:35.612964 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:35.613019 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:35.613621 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:35.614654 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:35.615652 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:35.615979 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:35.616037 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:35.617410 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:35.617411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:35.617412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:35.617418 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:35.618983 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:35.619343 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:35.619344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:35.619344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:35.619398 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:35.619581 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:35.619581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:35.619582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:35.619689 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:35.619770 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:35.619899 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 19:04:35.619965 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 19:04:35.629982 22371 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:35.630602 22371 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:35.639840 22371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ecb7b13a9314c1ca942fdbd4e1d6db9: Generated new TSK 0
I20260925 19:04:35.640043 22371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:35.645761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 22290
2026-09-25T19:04:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:35 dist-test-slave-49bz krb5kdc[22389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:35 dist-test-slave-49bz krb5kdc[22389](info): set up 2 sockets
Sep 25 19:04:35 dist-test-slave-49bz krb5kdc[22389](info): commencing operation
krb5kdc: starting...
W20260925 19:04:36.662212 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:36 dist-test-slave-49bz krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:36.776430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36717
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:36.853250 22402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:36.853550 22402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:36.853586 22402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:36.854821 22402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:36.854897 22402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:36.854919 22402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:36.854941 22402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:36.854972 22402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:36.856415 22402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:36.856693 22402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:36.856914 22402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:36.861871 22408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:36.862039 22410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:36.861958 22402 server_base.cc:1034] running on GCE node
W20260925 19:04:36.861857 22407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:36.862411 22402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:36.862653 22402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:36.863816 22402 hybrid_clock.cc:648] HybridClock initialized: now 1790363076863800 us; error 46 us; skew 500 ppm
Sep 25 19:04:36 dist-test-slave-49bz krb5kdc[22389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:36.866184 22402 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:36.866601 22402 webserver.cc:460] Webserver started at http://127.0.98.62:44351/ using document root <none> and password file <none>
I20260925 19:04:36.866824 22402 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:36.866889 22402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:36.867031 22402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:36.868062 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "8bc458db5236497da0d7d216a558c22f"
format_stamp: "Formatted at 2026-09-25 19:04:36 on dist-test-slave-49bz"
server_key: "38b2fc59b147f3b72adfe684d08be0c3"
server_key_iv: "7e800204e75a426aa9ec626bca9e6434"
server_key_version: "encryptionkey@0"
I20260925 19:04:36.868408 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "8bc458db5236497da0d7d216a558c22f"
format_stamp: "Formatted at 2026-09-25 19:04:36 on dist-test-slave-49bz"
server_key: "38b2fc59b147f3b72adfe684d08be0c3"
server_key_iv: "7e800204e75a426aa9ec626bca9e6434"
server_key_version: "encryptionkey@0"
I20260925 19:04:36.869649 22402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:36.870326 22417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:36.870484 22402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:36.870572 22402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "8bc458db5236497da0d7d216a558c22f"
format_stamp: "Formatted at 2026-09-25 19:04:36 on dist-test-slave-49bz"
server_key: "38b2fc59b147f3b72adfe684d08be0c3"
server_key_iv: "7e800204e75a426aa9ec626bca9e6434"
server_key_version: "encryptionkey@0"
I20260925 19:04:36.870657 22402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:36.881932 22402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:36.883715 22402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:36.883865 22402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:36.887655 22402 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36717
I20260925 19:04:36.887689 22469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36717 every 8 connection(s)
I20260925 19:04:36.888021 22402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:36.888832 22470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:36.890458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 22402
I20260925 19:04:36.890590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:36.890764 392 external_mini_cluster.cc:1428] Setting key 1298d6739b6dd99d00f5ccaefaa1cae9
I20260925 19:04:36.891292 22470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f: Bootstrap starting.
I20260925 19:04:36.891880 22470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:36.892251 22470 log.cc:824] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:36.892920 22470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f: No bootstrap required, opened a new log
I20260925 19:04:36.895215 22470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc458db5236497da0d7d216a558c22f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36717 } }
I20260925 19:04:36.895403 22470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:36.895447 22470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc458db5236497da0d7d216a558c22f, State: Initialized, Role: FOLLOWER
I20260925 19:04:36.895593 22470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc458db5236497da0d7d216a558c22f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36717 } }
I20260925 19:04:36.895684 22470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:36.895725 22470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:36.895771 22470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:36.896207 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:36.891259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39395 (local address 127.0.98.62:36717)
0925 19:04:36.891424 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:36.891433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:36.891463 (+ 30us) server_negotiation.cc:407] Connection header received
0925 19:04:36.891850 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:36.891856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:36.891898 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:36.892042 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:36.892283 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:36.893228 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:36.894335 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:36.894721 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:36.894795 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:36.895518 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:36.895521 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:36.895522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:36.895724 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:36.895762 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:36.895776 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:36.895873 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:36.895950 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 19:04:36.896011 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260925 19:04:36.896505 22470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc458db5236497da0d7d216a558c22f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36717 } }
I20260925 19:04:36.896605 22470 leader_election.cc:302] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bc458db5236497da0d7d216a558c22f; no voters:
I20260925 19:04:36.896752 22470 leader_election.cc:288] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:36.896771 22475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:36.896912 22475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [term 1 LEADER]: Becoming Leader. State: Replica: 8bc458db5236497da0d7d216a558c22f, State: Running, Role: LEADER
I20260925 19:04:36.897171 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:36.897140 22475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc458db5236497da0d7d216a558c22f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36717 } }
I20260925 19:04:36.897029 22470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:36.897490 22476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bc458db5236497da0d7d216a558c22f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc458db5236497da0d7d216a558c22f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36717 } } }
I20260925 19:04:36.897511 22477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc458db5236497da0d7d216a558c22f. Latest consensus state: current_term: 1 leader_uuid: "8bc458db5236497da0d7d216a558c22f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc458db5236497da0d7d216a558c22f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36717 } } }
I20260925 19:04:36.897565 22476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:36.897572 22477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:36.897830 22479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:36.898447 22479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:36.899942 22479 catalog_manager.cc:1347] Generated new cluster ID: 6b65ebed398543d89c764dba4c91e510
I20260925 19:04:36.899996 22479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:36.908185 22479 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:36.908813 22479 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:36.909317 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:36.898712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48689 (local address 127.0.98.62:36717)
0925 19:04:36.898747 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:36.898749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:36.906450 (+ 7701us) server_negotiation.cc:407] Connection header received
0925 19:04:36.906525 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:36.906526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:36.906537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:36.906608 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:36.906749 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:36.907204 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:36.908091 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:36.908370 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:36.908402 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:36.909052 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:36.909054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:36.909054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:36.909058 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:36.909071 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:36.909075 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:36.909129 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:36.909196 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 19:04:36.909228 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:04:36.914111 22479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bc458db5236497da0d7d216a558c22f: Generated new TSK 0
I20260925 19:04:36.914284 22479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:36.917140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 22402
2026-09-25T19:04:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1270 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:36 dist-test-slave-49bz krb5kdc[22501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:36 dist-test-slave-49bz krb5kdc[22501](info): set up 2 sockets
Sep 25 19:04:36 dist-test-slave-49bz krb5kdc[22501](info): commencing operation
krb5kdc: starting...
W20260925 19:04:37.944002 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:38 dist-test-slave-49bz krb5kdc[22501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:38.061520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37947
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:38.138558 22514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:38.138787 22514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:38.138823 22514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:38.139959 22514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:38.140027 22514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:38.140058 22514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:38.140085 22514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:38.140117 22514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:38.141583 22514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:38.141841 22514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:38.142060 22514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:38.146737 22520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:38.146721 22522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:38.146721 22519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:38.146988 22514 server_base.cc:1034] running on GCE node
I20260925 19:04:38.147260 22514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:38.147542 22514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:38.148681 22514 hybrid_clock.cc:648] HybridClock initialized: now 1790363078148667 us; error 27 us; skew 500 ppm
Sep 25 19:04:38 dist-test-slave-49bz krb5kdc[22501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:38.156430 22514 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:38.156843 22514 webserver.cc:460] Webserver started at http://127.0.98.62:35429/ using document root <none> and password file <none>
I20260925 19:04:38.157068 22514 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:38.157135 22514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:38.157279 22514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:38.158322 22514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "31d5db5f8e2443a48c46e2b224156bb4"
format_stamp: "Formatted at 2026-09-25 19:04:38 on dist-test-slave-49bz"
server_key: "57cfcc6cbf4299d873b489a46e37f4ae"
server_key_iv: "9f591ffe6531a505d41b77ba9cffa66e"
server_key_version: "encryptionkey@0"
I20260925 19:04:38.158667 22514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "31d5db5f8e2443a48c46e2b224156bb4"
format_stamp: "Formatted at 2026-09-25 19:04:38 on dist-test-slave-49bz"
server_key: "57cfcc6cbf4299d873b489a46e37f4ae"
server_key_iv: "9f591ffe6531a505d41b77ba9cffa66e"
server_key_version: "encryptionkey@0"
I20260925 19:04:38.159896 22514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:38.160593 22529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:38.160761 22514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:38.160857 22514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "31d5db5f8e2443a48c46e2b224156bb4"
format_stamp: "Formatted at 2026-09-25 19:04:38 on dist-test-slave-49bz"
server_key: "57cfcc6cbf4299d873b489a46e37f4ae"
server_key_iv: "9f591ffe6531a505d41b77ba9cffa66e"
server_key_version: "encryptionkey@0"
I20260925 19:04:38.160949 22514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:38.164206 22514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:38.166528 22514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:38.166680 22514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:38.170403 22514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37947
I20260925 19:04:38.170447 22581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37947 every 8 connection(s)
I20260925 19:04:38.170750 22514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:38.171712 22582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:38.173785 22582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4: Bootstrap starting.
I20260925 19:04:38.174372 22582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:38.174671 22582 log.cc:824] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:38.175366 22582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4: No bootstrap required, opened a new log
I20260925 19:04:38.175364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 22514
I20260925 19:04:38.175467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:38.175643 392 external_mini_cluster.cc:1428] Setting key 7de5e6469568b3f2599ea38e441dde84
I20260925 19:04:38.177163 22582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5db5f8e2443a48c46e2b224156bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37947 } }
I20260925 19:04:38.177311 22582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 19:04:38.177373 22582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5db5f8e2443a48c46e2b224156bb4, State: Initialized, Role: FOLLOWER
I20260925 19:04:38.177551 22582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5db5f8e2443a48c46e2b224156bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37947 } }
I20260925 19:04:38.177661 22582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 19:04:38.177697 22582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 19:04:38.177783 22582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:38.178445 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:38.176139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59871 (local address 127.0.98.62:37947)
0925 19:04:38.176324 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:38.176350 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:38.176382 (+ 32us) server_negotiation.cc:407] Connection header received
0925 19:04:38.177152 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:38.177158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:38.177182 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:38.177308 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:38.177957 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:38.177960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:38.177961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:38.177965 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:38.178007 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:38.178051 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:38.178128 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:38.178197 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 19:04:38.178236 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260925 19:04:38.178922 22582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5db5f8e2443a48c46e2b224156bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37947 } }
I20260925 19:04:38.179070 22582 leader_election.cc:302] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d5db5f8e2443a48c46e2b224156bb4; no voters:
I20260925 19:04:38.179237 22582 leader_election.cc:288] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 19:04:38.179270 22587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:38.179382 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:38.179467 22582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:38.179489 22587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 31d5db5f8e2443a48c46e2b224156bb4, State: Running, Role: LEADER
I20260925 19:04:38.181525 22587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5db5f8e2443a48c46e2b224156bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37947 } }
I20260925 19:04:38.182050 22588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d5db5f8e2443a48c46e2b224156bb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5db5f8e2443a48c46e2b224156bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37947 } } }
I20260925 19:04:38.182129 22588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:38.182212 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:38.181091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40317 (local address 127.0.98.62:37947)
0925 19:04:38.181281 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:38.181283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:38.181292 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:38.181313 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:38.181314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:38.181319 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:38.181386 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:38.181951 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:38.181953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:38.181953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 19:04:38.181957 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:38.181970 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 19:04:38.181974 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:38.182025 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:38.182088 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 19:04:38.182103 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260925 19:04:38.182291 22588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d5db5f8e2443a48c46e2b224156bb4. Latest consensus state: current_term: 1 leader_uuid: "31d5db5f8e2443a48c46e2b224156bb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5db5f8e2443a48c46e2b224156bb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37947 } } }
I20260925 19:04:38.182336 22588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:38.182367 22606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:38.182845 22606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:38.184723 22606 catalog_manager.cc:1347] Generated new cluster ID: 927f4cbc79e54baa9f82a6d00784bd88
I20260925 19:04:38.184760 22606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:38.193918 22606 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:38.194059 22606 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:38.198443 22606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31d5db5f8e2443a48c46e2b224156bb4: Generated new TSK 0
I20260925 19:04:38.198616 22606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:38.206318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 22514
2026-09-25T19:04:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1289 ms)
[----------] 18 tests from AuthTokenIssuingTest (23297 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:38 dist-test-slave-49bz krb5kdc[22613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:38 dist-test-slave-49bz krb5kdc[22613](info): set up 2 sockets
Sep 25 19:04:38 dist-test-slave-49bz krb5kdc[22613](info): commencing operation
krb5kdc: starting...
W20260925 19:04:39.222275 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:39.339221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39213
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39213,127.0.98.61:38475,127.0.98.60:41403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:39.418732 22626 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:04:39.418977 22626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:39.419008 22626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:39.419049 22626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:39.420127 22626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:39.420193 22626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:39.420217 22626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:39.420238 22626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:39.420264 22626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:39.421731 22626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39213,127.0.98.61:38475,127.0.98.60:41403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:39.422014 22626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:39.422236 22626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:39.426949 22632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:39.426937 22631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:39.426939 22634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:39.427265 22626 server_base.cc:1034] running on GCE node
I20260925 19:04:39.427505 22626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:39.427834 22626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:39.429020 22626 hybrid_clock.cc:648] HybridClock initialized: now 1790363079428994 us; error 58 us; skew 500 ppm
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:39.431623 22626 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:39.432060 22626 webserver.cc:460] Webserver started at http://127.0.98.62:42853/ using document root <none> and password file <none>
I20260925 19:04:39.432287 22626 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:39.432355 22626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:39.432518 22626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:39.433580 22626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "029f65052f9149c8a5c1a76da1ead36f"
format_stamp: "Formatted at 2026-09-25 19:04:39 on dist-test-slave-49bz"
server_key: "134bf52c51d00c2598ed647864f7639c"
server_key_iv: "8be4f38e6d67a3a267bbc6f83c85e0ae"
server_key_version: "encryptionkey@0"
I20260925 19:04:39.433945 22626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "029f65052f9149c8a5c1a76da1ead36f"
format_stamp: "Formatted at 2026-09-25 19:04:39 on dist-test-slave-49bz"
server_key: "134bf52c51d00c2598ed647864f7639c"
server_key_iv: "8be4f38e6d67a3a267bbc6f83c85e0ae"
server_key_version: "encryptionkey@0"
I20260925 19:04:39.435402 22626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:39.436151 22641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:39.436337 22626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:39.436455 22626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "029f65052f9149c8a5c1a76da1ead36f"
format_stamp: "Formatted at 2026-09-25 19:04:39 on dist-test-slave-49bz"
server_key: "134bf52c51d00c2598ed647864f7639c"
server_key_iv: "8be4f38e6d67a3a267bbc6f83c85e0ae"
server_key_version: "encryptionkey@0"
I20260925 19:04:39.436540 22626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:39.448576 22626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:39.450917 22626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:39.451077 22626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:39.454854 22626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39213
I20260925 19:04:39.454890 22693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39213 every 8 connection(s)
I20260925 19:04:39.455283 22626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:39.456362 22694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:39.457828 22694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:39.463804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 22626
I20260925 19:04:39.463898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:39.464087 392 external_mini_cluster.cc:1428] Setting key 3961df067bfa260fb2c74e524edd49b6
I20260925 19:04:39.467696 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.458570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39213 (local address 127.0.0.1:44932)
0925 19:04:39.458759 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.458800 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:39.459280 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:39.459598 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:39.459601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:39.459889 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:39.460037 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.460041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.461063 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.461064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.461742 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.461743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.462301 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.462302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.462371 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.462841 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:39.462849 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:39.464276 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:39.466109 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.466116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.466118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.466391 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.466642 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.466642 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.466643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.466734 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.467113 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:39.467117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:39.467179 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 19:04:39.467285 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 19:04:39.467514 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":91,"thread_start_us":83,"threads_started":1}
I20260925 19:04:39.467700 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.458609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.0.98.62:39213)
0925 19:04:39.458766 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.458776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.458917 (+ 141us) server_negotiation.cc:407] Connection header received
0925 19:04:39.459464 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.459466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.459486 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.459594 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.460139 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.460963 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.461846 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.462172 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.462308 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.464376 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.464384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.464384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.464394 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.466004 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.466484 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.466487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.466488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.466572 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.466822 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.466823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.466824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.467035 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.467124 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.467316 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.467451 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260925 19:04:39.468456 22694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:39.468863 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.468711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38475 (local address 127.0.0.1:52172)
0925 19:04:39.468763 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.468841 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:04:39.469025 22642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38475: connect: Connection refused (error 111)
I20260925 19:04:39.469478 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.469381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38475 (local address 127.0.0.1:52178)
0925 19:04:39.469437 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.469462 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 19:04:39.469600 22694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38475: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38475: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 19:04:39.488622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38475
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39213,127.0.98.61:38475,127.0.98.60:41403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:04:39.518828 22694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } attempt: 1
I20260925 19:04:39.519362 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.519193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38475 (local address 127.0.0.1:52182)
0925 19:04:39.519264 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.519318 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 19:04:39.519749 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.519655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38475 (local address 127.0.0.1:52186)
0925 19:04:39.519701 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.519730 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:04:39.519873 22694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38475: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38475: connect: Connection refused (error 111)
W20260925 19:04:39.569048 22701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:04:39.569247 22701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:39.569264 22701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:39.569283 22701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:39.570420 22701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:39.570461 22701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:39.570475 22701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:39.570514 22701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:39.570540 22701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:39.572016 22701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39213,127.0.98.61:38475,127.0.98.60:41403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:39.572274 22701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:39.572502 22701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:39.577355 22709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:39.577406 22707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:39.577543 22706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:39.577386 22701 server_base.cc:1034] running on GCE node
I20260925 19:04:39.577831 22701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:39.578070 22701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:39.579233 22701 hybrid_clock.cc:648] HybridClock initialized: now 1790363079579201 us; error 43 us; skew 500 ppm
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:39.581776 22701 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 19:04:39.582219 22701 webserver.cc:460] Webserver started at http://127.0.98.61:37859/ using document root <none> and password file <none>
I20260925 19:04:39.582474 22701 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:39.582544 22701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:39.582692 22701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:39.583740 22701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/data/instance:
uuid: "6ca733d1f8f6470ead3941861045c7b2"
format_stamp: "Formatted at 2026-09-25 19:04:39 on dist-test-slave-49bz"
server_key: "de1ab1feb6cb2697ca6722f91b4be962"
server_key_iv: "74e2ca03f012f1ddee636f17ff1ae78a"
server_key_version: "encryptionkey@0"
I20260925 19:04:39.584105 22701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/wal/instance:
uuid: "6ca733d1f8f6470ead3941861045c7b2"
format_stamp: "Formatted at 2026-09-25 19:04:39 on dist-test-slave-49bz"
server_key: "de1ab1feb6cb2697ca6722f91b4be962"
server_key_iv: "74e2ca03f012f1ddee636f17ff1ae78a"
server_key_version: "encryptionkey@0"
I20260925 19:04:39.585415 22701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:39.586129 22716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:39.586351 22701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 19:04:39.586452 22701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/wal
uuid: "6ca733d1f8f6470ead3941861045c7b2"
format_stamp: "Formatted at 2026-09-25 19:04:39 on dist-test-slave-49bz"
server_key: "de1ab1feb6cb2697ca6722f91b4be962"
server_key_iv: "74e2ca03f012f1ddee636f17ff1ae78a"
server_key_version: "encryptionkey@0"
I20260925 19:04:39.586536 22701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:39.588054 22694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } attempt: 2
I20260925 19:04:39.588423 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.588302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38475 (local address 127.0.0.1:52194)
0925 19:04:39.588357 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.588387 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:04:39.588762 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.588668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38475 (local address 127.0.0.1:52196)
0925 19:04:39.588719 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.588742 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 19:04:39.588853 22694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38475: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38475: connect: Connection refused (error 111)
I20260925 19:04:39.597234 22701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:39.599558 22701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:39.599711 22701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:39.603796 22701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38475
I20260925 19:04:39.603829 22768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38475 every 8 connection(s)
I20260925 19:04:39.604205 22701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
I20260925 19:04:39.605116 22769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:39.606629 22769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:39.613353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 22701
I20260925 19:04:39.613454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-1/wal/instance
I20260925 19:04:39.613628 392 external_mini_cluster.cc:1428] Setting key f4309bd49ce10cbde04d08d33161c348
I20260925 19:04:39.615012 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.607255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.0.98.62:39213)
0925 19:04:39.607335 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.607337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.607486 (+ 149us) server_negotiation.cc:407] Connection header received
0925 19:04:39.607910 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.607911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.607922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.607971 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.608528 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.608980 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.609869 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.610065 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.610114 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.612130 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.612132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.612132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.612138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.613635 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.614035 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.614036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.614037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.614097 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.614388 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.614389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.614389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.614503 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.614577 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.614841 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.614923 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 19:04:39.615185 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.607204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39213 (local address 127.0.0.1:44934)
0925 19:04:39.607373 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.607406 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:39.607823 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:39.608022 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:39.608025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:39.608257 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:39.608458 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.608462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.609055 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.609055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.609793 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.609794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.610151 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.610152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.610244 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.610745 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:39.610752 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:39.612047 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:39.613727 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.613735 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.613737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.613959 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.614160 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.614161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.614162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.614319 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.614569 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:39.614575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:39.614710 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 19:04:39.614809 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 19:04:39.614967 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260925 19:04:39.615721 22769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260925 19:04:39.622941 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.615971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38475 (local address 127.0.0.1:52208)
0925 19:04:39.616017 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.616028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:39.616063 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:39.616282 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:39.616283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:39.616411 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:39.616450 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.616451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.617291 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.617292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.617774 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.617775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.618234 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.618235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.618342 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.618738 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:39.618743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:39.620136 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:39.621909 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.621910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.621911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.622084 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.622288 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.622289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.622289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.622349 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.622577 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:39.622578 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:39.622615 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 19:04:39.622687 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 19:04:39.622810 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 19:04:39.622963 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.616038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.0.98.61:38475)
0925 19:04:39.616161 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.616165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.616178 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:04:39.616192 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.616193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.616202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.616269 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.616518 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.617200 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.617864 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.618152 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.618237 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.620217 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.620224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.620224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.620230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.621840 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.622150 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.622152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.622152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.622199 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.622397 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.622398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.622398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.622523 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.622605 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.622734 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.622863 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260925 19:04:39.623622 22769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:39.623955 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.623840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48294)
0925 19:04:39.623885 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.623928 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 19:04:39.624076 22720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
I20260925 19:04:39.624466 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.624379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48310)
0925 19:04:39.624425 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.624447 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:04:39.624567 22769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 19:04:39.637682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41403
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39213,127.0.98.61:38475,127.0.98.60:41403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:04:39.673787 22769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } attempt: 1
I20260925 19:04:39.674304 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.674148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48312)
0925 19:04:39.674208 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.674267 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:04:39.674690 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.674601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48316)
0925 19:04:39.674645 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.674671 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 19:04:39.674782 22769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
I20260925 19:04:39.679993 22694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } attempt: 3
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 19:04:39.687527 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.680349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38475 (local address 127.0.0.1:52222)
0925 19:04:39.680410 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.680422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:39.680473 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:39.680646 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:39.680647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:39.680812 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:39.680900 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.680902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.681827 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.681828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.682415 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.682416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.682877 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.682878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.682934 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.683419 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:39.683424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:39.684808 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:39.686463 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.686464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.686464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.686738 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.687014 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.687015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.687015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.687070 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.687316 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:39.687316 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:39.687352 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 19:04:39.687397 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 19:04:39.687466 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 19:04:39.687569 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.680451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.0.98.61:38475)
0925 19:04:39.680508 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.680510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.680520 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:04:39.680548 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.680549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.680561 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.680638 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.680975 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.681722 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.682485 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.682799 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.682882 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.684889 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.684890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.684890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.684898 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.686371 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.686863 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.686864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.686864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.686945 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.687128 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.687129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.687129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.687256 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.687319 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.687418 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.687490 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:04:39.688122 22694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:39.688462 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.688355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48322)
0925 19:04:39.688401 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.688435 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 19:04:39.688828 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.688746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48338)
0925 19:04:39.688784 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.688810 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 19:04:39.688912 22694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
I20260925 19:04:39.720083 22694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } attempt: 1
W20260925 19:04:39.720016 22776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:04:39.720214 22776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:39.720232 22776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:39.720278 22776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 19:04:39.720585 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.720436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48354)
0925 19:04:39.720507 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.720551 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 19:04:39.720958 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.720861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48366)
0925 19:04:39.720906 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.720937 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:04:39.721057 22694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
W20260925 19:04:39.721735 22776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:39.721776 22776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:39.721791 22776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:39.721837 22776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:39.721859 22776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:39.723408 22776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39213,127.0.98.61:38475,127.0.98.60:41403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:39.723670 22776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:39.723919 22776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:39.728611 22781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:39.728739 22784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:39.728621 22782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:39.728901 22776 server_base.cc:1034] running on GCE node
I20260925 19:04:39.729167 22776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:39.729487 22776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:39.730611 22776 hybrid_clock.cc:648] HybridClock initialized: now 1790363079730589 us; error 32 us; skew 500 ppm
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:39.733073 22776 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 19:04:39.733500 22776 webserver.cc:460] Webserver started at http://127.0.98.60:36195/ using document root <none> and password file <none>
I20260925 19:04:39.733752 22776 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:39.733832 22776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:39.733985 22776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:39.735146 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/data/instance:
uuid: "cae1fe31df264002b8114b6832697ecf"
format_stamp: "Formatted at 2026-09-25 19:04:39 on dist-test-slave-49bz"
server_key: "477a18492d377dd46d507908e5ef3d69"
server_key_iv: "7f40f1de1fee3e2a0f6bcf7107edf696"
server_key_version: "encryptionkey@0"
I20260925 19:04:39.735507 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/wal/instance:
uuid: "cae1fe31df264002b8114b6832697ecf"
format_stamp: "Formatted at 2026-09-25 19:04:39 on dist-test-slave-49bz"
server_key: "477a18492d377dd46d507908e5ef3d69"
server_key_iv: "7f40f1de1fee3e2a0f6bcf7107edf696"
server_key_version: "encryptionkey@0"
I20260925 19:04:39.736874 22776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:39.737565 22791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:39.737736 22776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:39.737830 22776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/wal
uuid: "cae1fe31df264002b8114b6832697ecf"
format_stamp: "Formatted at 2026-09-25 19:04:39 on dist-test-slave-49bz"
server_key: "477a18492d377dd46d507908e5ef3d69"
server_key_iv: "7f40f1de1fee3e2a0f6bcf7107edf696"
server_key_version: "encryptionkey@0"
I20260925 19:04:39.737926 22776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:39.742899 22769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } attempt: 2
I20260925 19:04:39.743283 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.743155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48376)
0925 19:04:39.743222 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.743257 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 19:04:39.743619 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.743537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48390)
0925 19:04:39.743578 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.743600 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 19:04:39.743711 22769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41403: connect: Connection refused (error 111)
I20260925 19:04:39.749411 22776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:39.751806 22776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:39.752007 22776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:39.756067 22776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41403
I20260925 19:04:39.756100 22843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41403 every 8 connection(s)
I20260925 19:04:39.756455 22776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
I20260925 19:04:39.757401 22844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:39.758956 22844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:39.762305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 22776
I20260925 19:04:39.762427 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790362794046639-392-0/minicluster-data/master-2/wal/instance
I20260925 19:04:39.762602 392 external_mini_cluster.cc:1428] Setting key 6d503263071d57fe477a5322cfc51743
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 19:04:39.768882 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.759597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39213 (local address 127.0.0.1:44944)
0925 19:04:39.759776 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.759813 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:39.760271 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:39.760551 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:39.760554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:39.760809 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:39.761004 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.761008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.761747 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.761748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.762650 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.762652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.763173 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.763174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.763272 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.763771 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:39.763781 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:39.765325 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:39.767272 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.767277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.767279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.767507 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.767761 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.767762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.767762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.767918 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.768175 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:39.768180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:39.768331 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 19:04:39.768457 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 19:04:39.768624 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260925 19:04:39.769075 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.759613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.0.98.62:39213)
0925 19:04:39.759675 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.759679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.759895 (+ 216us) server_negotiation.cc:407] Connection header received
0925 19:04:39.760379 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.760380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.760391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.760451 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.761110 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.761667 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.762739 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.763006 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.763099 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.765423 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.765424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.765424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.765432 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.767194 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.767590 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.767591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.767592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.767654 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.767989 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.767990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.767991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.768097 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.768162 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.768914 (+ 752us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.768992 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:04:39.769759 22844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:39.772014 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.763009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.0.98.60:41403)
0925 19:04:39.763316 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.763320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.763335 (+ 15us) server_negotiation.cc:407] Connection header received
0925 19:04:39.763352 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.763353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.763368 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.763421 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.764229 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.764913 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.765730 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.766045 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.766100 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.768887 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.768893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.768894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.768900 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.770744 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.771162 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.771164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.771164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.771220 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.771408 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.771409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.771409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.771608 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.771747 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.771811 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.771906 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":281,"thread_start_us":93,"threads_started":1}
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 19:04:39.777179 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.770088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.0.98.61:38475)
0925 19:04:39.770128 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.770130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.770171 (+ 41us) server_negotiation.cc:407] Connection header received
0925 19:04:39.770241 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.770242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.770297 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.770342 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.770689 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.771437 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.772368 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.772621 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.772670 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.774585 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.774586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.774587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.774594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.776167 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.776463 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.776465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.776465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.776522 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.776733 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.776733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.776733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.776843 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.776896 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.777066 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.777120 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:39.777240 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.770020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38475 (local address 127.0.0.1:52228)
0925 19:04:39.770130 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.770139 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:39.770184 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:39.770368 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:39.770369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:39.770581 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:39.770617 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.770618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.771506 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.771507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.772281 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.772284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.772828 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.772829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.772895 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.773290 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:39.773294 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:39.774504 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:39.776245 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.776246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.776246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.776396 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.776607 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.776608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.776609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.776670 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.776914 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:39.776914 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:39.776958 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 19:04:39.777049 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 19:04:39.777139 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":5}
I20260925 19:04:39.777603 22844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:39.780578 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.772831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.0.98.62:39213)
0925 19:04:39.772890 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.772891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.772900 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:39.772926 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.772926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.772936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.772999 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.773573 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.774117 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.775002 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.775232 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.775283 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.777756 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.777757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.777758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.777765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.779411 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.779763 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.779764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.779764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.779816 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.780026 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.780027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.780028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.780134 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.780302 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.780337 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.780414 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 19:04:39.785379 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.777877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48396)
0925 19:04:39.777963 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.777972 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:39.778019 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:39.778135 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:39.778136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:39.778306 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:39.778339 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.778341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.779250 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.779251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.779825 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.779827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.780196 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.780196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.780242 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.780606 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:39.780611 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:39.781968 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:39.784362 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.784363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.784363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.784515 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.784709 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.784710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.784710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.784787 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.785022 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:39.785023 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:39.785066 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 19:04:39.785152 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:04:39.785242 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 19:04:39.785439 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.777930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.0.98.60:41403)
0925 19:04:39.777968 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.777969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.778006 (+ 37us) server_negotiation.cc:407] Connection header received
0925 19:04:39.778072 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.778073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.778080 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.778116 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.778422 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.779167 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.779923 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.780117 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.780196 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.782063 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.782065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.782065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.782071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.784267 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.784584 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.784585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.784586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.784643 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.784856 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.784857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.784857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.784953 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.785004 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.785177 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.785281 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":18}
I20260925 19:04:39.786875 22844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf: Bootstrap starting.
I20260925 19:04:39.787779 22844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:39.788079 22844 log.cc:824] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:39.788455 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.781255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.0.98.61:38475)
0925 19:04:39.781306 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.781307 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.781315 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:39.781335 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.781336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.781344 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.781416 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.781887 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.782477 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.783103 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.783286 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.783335 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.785807 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.785808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.785809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.785816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.787367 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.787723 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.787724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.787724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.787779 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.787976 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.787977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.787977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.788086 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.788246 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.788281 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.788359 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 19:04:39.788940 22844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf: No bootstrap required, opened a new log
I20260925 19:04:39.789940 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:39.790642 22844 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } }
I20260925 19:04:39.790829 22844 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae1fe31df264002b8114b6832697ecf, State: Initialized, Role: FOLLOWER
I20260925 19:04:39.790930 22844 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } }
I20260925 19:04:39.791251 22844 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:39.791234 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } } }
I20260925 19:04:39.791392 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:39.791098 22668 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "029f65052f9149c8a5c1a76da1ead36f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44954
W20260925 19:04:39.792963 22861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:04:39.793013 22861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:04:39.793294 22776 master_runner.cc:186] Error getting master registration for 127.0.98.62:39213: OK, instance_id { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" instance_seqno: 1790363079453724 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:04:39.796146 22776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:04:39.796171 22694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } attempt: 2
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 19:04:39.803347 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.796460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48402 (local address 127.0.98.60:41403)
0925 19:04:39.796502 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.796504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.796513 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:39.796568 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.796569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.796578 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.796629 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.796911 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.797366 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.798026 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.798225 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.798358 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.800512 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.800513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.800514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.800520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.802097 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.802495 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.802496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.802496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.802550 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.802755 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.802756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.802757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.802913 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.802977 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.803171 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.803256 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 19:04:39.803372 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.796421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48402)
0925 19:04:39.796459 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.796469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:39.796510 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:39.796633 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:39.796634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:39.796786 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:39.796828 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.796830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.797445 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.797446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.797946 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.797948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.798358 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.798359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.798414 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.798882 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:39.798886 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:39.800415 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:39.802184 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.802185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.802185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.802424 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.802621 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.802622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.802622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.802689 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.802984 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:39.802986 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:39.803039 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 19:04:39.803129 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 19:04:39.803265 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 19:04:39.804633 22694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f: Bootstrap starting.
I20260925 19:04:39.805279 22694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:39.805579 22694 log.cc:824] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:39.806399 22694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f: No bootstrap required, opened a new log
I20260925 19:04:39.807838 22694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } }
I20260925 19:04:39.807974 22694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 029f65052f9149c8a5c1a76da1ead36f, State: Initialized, Role: FOLLOWER
I20260925 19:04:39.808059 22694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } }
I20260925 19:04:39.808295 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } } }
I20260925 19:04:39.808357 22694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:39.808375 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:39.808483 22668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 19:04:39.808561 22668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:39.809293 22668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } }
I20260925 19:04:39.809749 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "029f65052f9149c8a5c1a76da1ead36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cae1fe31df264002b8114b6832697ecf"
I20260925 19:04:39.809880 22818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [term 0 FOLLOWER]: Advancing to term 1
W20260925 19:04:39.810678 22875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:04:39.810730 22875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:04:39.810747 22818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 029f65052f9149c8a5c1a76da1ead36f in term 1.
I20260925 19:04:39.810971 22645 leader_election.cc:302] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 029f65052f9149c8a5c1a76da1ead36f, cae1fe31df264002b8114b6832697ecf; no voters:
I20260925 19:04:39.811088 22864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:39.811270 22864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [term 1 LEADER]: Becoming Leader. State: Replica: 029f65052f9149c8a5c1a76da1ead36f, State: Running, Role: LEADER
I20260925 19:04:39.811353 22864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } }
I20260925 19:04:39.811098 22626 master_runner.cc:186] Error getting master registration for 127.0.98.61:38475: OK, instance_id { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" instance_seqno: 1790363079602627 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:04:39.814005 22626 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:04:39.814075 22668 leader_election.cc:288] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [CANDIDATE]: Term 1 election: Requested vote from peers 6ca733d1f8f6470ead3941861045c7b2 (127.0.98.61:38475), cae1fe31df264002b8114b6832697ecf (127.0.98.60:41403)
I20260925 19:04:39.811746 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 029f65052f9149c8a5c1a76da1ead36f. Latest consensus state: current_term: 1 leader_uuid: "029f65052f9149c8a5c1a76da1ead36f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } } }
I20260925 19:04:39.814224 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:39.815896 22879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:39.816093 22879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:39.818658 22818 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [term 1 FOLLOWER]: Refusing update from remote peer 029f65052f9149c8a5c1a76da1ead36f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:39.819182 22877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:39.820519 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 029f65052f9149c8a5c1a76da1ead36f. Latest consensus state: current_term: 1 leader_uuid: "029f65052f9149c8a5c1a76da1ead36f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } } }
I20260925 19:04:39.820617 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:39.826223 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.815210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.0.98.62:39213)
0925 19:04:39.815256 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.815258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.815759 (+ 501us) server_negotiation.cc:407] Connection header received
0925 19:04:39.815833 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.815834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.815860 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.815922 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.817560 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.819135 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.820580 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.820868 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.820914 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.822556 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.822557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.822558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.822565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.824347 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.825191 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.825192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.825192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.825247 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.825484 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.825486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.825486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.825638 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.825744 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.825995 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.826088 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:04:39.833822 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.815389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48406 (local address 127.0.98.60:41403)
0925 19:04:39.815425 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.815426 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.821354 (+ 5928us) server_negotiation.cc:407] Connection header received
0925 19:04:39.821456 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.821457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.821467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.821496 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.823123 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.823853 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.824861 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.825162 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.825227 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.830290 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.830291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.830292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.830370 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.832026 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.832728 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.832729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.832730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.832788 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.833101 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.833101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.833102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.833242 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.833323 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.833647 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.833725 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:04:39.834841 22769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } attempt: 3
I20260925 19:04:39.836585 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.815489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.0.98.61:38475)
0925 19:04:39.815527 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.815528 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.826259 (+ 10731us) server_negotiation.cc:407] Connection header received
0925 19:04:39.826401 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.826403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.826414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.826445 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.827266 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.827994 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.829963 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.830462 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.830539 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.833273 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.833275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.833275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.833282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.834938 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.835439 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.835440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.835441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.835502 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.836037 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.836038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.836039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.836168 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.836256 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.836426 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.836499 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:04:39.837137 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "029f65052f9149c8a5c1a76da1ead36f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } } }
I20260925 19:04:39.837229 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:39.837314 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "029f65052f9149c8a5c1a76da1ead36f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } } }
I20260925 19:04:39.837395 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:39.837386 22888 mvcc.cc:205] Tried to move back new op lower bound from 7333327174932733952 to 7333327174908100608. Current Snapshot: MvccSnapshot[applied={T|T < 7333327174932733952}]
I20260925 19:04:39.837813 22887 mvcc.cc:205] Tried to move back new op lower bound from 7333327174932733952 to 7333327174908100608. Current Snapshot: MvccSnapshot[applied={T|T < 7333327174932733952}]
I20260925 19:04:39.838963 22879 catalog_manager.cc:1347] Generated new cluster ID: 95ec9f43b0d448a393512e589fe830cc
I20260925 19:04:39.839012 22879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 19:04:39 dist-test-slave-49bz krb5kdc[22613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 19:04:39.846586 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.835089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41403 (local address 127.0.0.1:48410)
0925 19:04:39.835166 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:39.835177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:39.835212 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:39.835478 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:39.835479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:39.835625 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:39.835666 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.835668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.836563 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.836565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.837280 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:39.837282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.837858 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.837859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:39.837906 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.838318 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:39.838324 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:39.839826 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:39.845039 (+ 5213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.845040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.845041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.845279 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.845592 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:39.845592 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:39.845593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:39.845660 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:39.846183 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:39.846184 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:39.846258 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 19:04:39.846380 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 19:04:39.846488 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 19:04:39.847849 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:39.835173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.0.98.60:41403)
0925 19:04:39.835220 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.835222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.835232 (+ 10us) server_negotiation.cc:407] Connection header received
0925 19:04:39.835267 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.835268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.835276 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.835332 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.835808 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.836478 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.837492 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:39.837739 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:39.837800 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:39.839935 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:39.839936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:39.839937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:39.839943 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:39.844899 (+ 4956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.845396 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.845397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.845397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.845482 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:39.845754 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:39.845755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:39.845755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:39.846011 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:39.847442 (+ 1431us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:39.847533 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 19:04:39.847697 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 19:04:39.849516 22769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2: Bootstrap starting.
I20260925 19:04:39.850180 22769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:39.850509 22769 log.cc:824] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:39.854039 22769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2: No bootstrap required, opened a new log
I20260925 19:04:39.856078 22769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } }
I20260925 19:04:39.856217 22769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca733d1f8f6470ead3941861045c7b2, State: Initialized, Role: FOLLOWER
I20260925 19:04:39.856295 22769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } }
I20260925 19:04:39.856640 22769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:39.856667 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "029f65052f9149c8a5c1a76da1ead36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ca733d1f8f6470ead3941861045c7b2"
I20260925 19:04:39.856796 22743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:39.857667 22743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 029f65052f9149c8a5c1a76da1ead36f in term 1.
I20260925 19:04:39.857865 22742 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [term 1 FOLLOWER]: Refusing update from remote peer 029f65052f9149c8a5c1a76da1ead36f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:39.858687 22879 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:39.858819 22877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:39.859371 22879 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:39.862347 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } } }
I20260925 19:04:39.862507 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:39.862592 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 029f65052f9149c8a5c1a76da1ead36f. Latest consensus state: current_term: 1 leader_uuid: "029f65052f9149c8a5c1a76da1ead36f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } } }
I20260925 19:04:39.862686 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:39.862752 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "029f65052f9149c8a5c1a76da1ead36f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } } }
I20260925 19:04:39.862851 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:39.867801 22906 mvcc.cc:205] Tried to move back new op lower bound from 7333327175079038976 to 7333327174908100608. Current Snapshot: MvccSnapshot[applied={T|T < 7333327174932733952}]
W20260925 19:04:39.868999 22904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:04:39.869062 22904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:04:39.874527 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "029f65052f9149c8a5c1a76da1ead36f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029f65052f9149c8a5c1a76da1ead36f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } peers { permanent_uuid: "6ca733d1f8f6470ead3941861045c7b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38475 } } peers { permanent_uuid: "cae1fe31df264002b8114b6832697ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41403 } } }
I20260925 19:04:39.874631 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:39.875763 22879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 029f65052f9149c8a5c1a76da1ead36f: Generated new TSK 0
I20260925 19:04:39.875962 22879 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 19:04:39.901532 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:39.900387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.0.98.60:41403)
0925 19:04:39.900434 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.900436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.900829 (+ 393us) server_negotiation.cc:407] Connection header received
0925 19:04:39.900943 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.900944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.900955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.900986 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.901462 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48426: BlockingRecv error: recv got EOF from 127.0.0.1:48426 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 19:04:39.903031 22697 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:39.902063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.0.98.62:39213)
0925 19:04:39.902106 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.902108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.902459 (+ 351us) server_negotiation.cc:407] Connection header received
0925 19:04:39.902589 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.902590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.902602 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.902632 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.902971 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44984: BlockingRecv error: recv got EOF from 127.0.0.1:44984 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 19:04:39.903967 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:39.902338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.0.98.61:38475)
0925 19:04:39.902378 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.902380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.903447 (+ 1067us) server_negotiation.cc:407] Connection header received
0925 19:04:39.903543 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.903545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.903557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.903587 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.903902 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52246: BlockingRecv error: recv got EOF from 127.0.0.1:52246 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 19:04:39.906358 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:39.905587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.0.98.61:38475)
0925 19:04:39.905622 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.905624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.905813 (+ 189us) server_negotiation.cc:407] Connection header received
0925 19:04:39.905917 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.905918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.905936 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.905965 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.906316 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52252: BlockingRecv error: recv got EOF from 127.0.0.1:52252 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:04:39.909588 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:39.908798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.0.98.61:38475)
0925 19:04:39.908830 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.908832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.909059 (+ 227us) server_negotiation.cc:407] Connection header received
0925 19:04:39.909150 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.909151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.909161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.909202 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.909547 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52254: BlockingRecv error: recv got EOF from 127.0.0.1:52254 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:04:39.923128 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:39.913076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.0.98.61:38475)
0925 19:04:39.913132 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.913134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.922489 (+ 9355us) server_negotiation.cc:407] Connection header received
0925 19:04:39.922602 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.922603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.922616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.922647 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.923072 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52268: BlockingRecv error: recv got EOF from 127.0.0.1:52268 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:04:39.929486 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:39.928485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.0.98.61:38475)
0925 19:04:39.928525 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.928527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.928836 (+ 309us) server_negotiation.cc:407] Connection header received
0925 19:04:39.928972 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.928973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.928985 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.929015 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.929436 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52284: BlockingRecv error: recv got EOF from 127.0.0.1:52284 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 19:04:39.939800 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:39.938811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.0.98.61:38475)
0925 19:04:39.938854 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.938856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.939200 (+ 344us) server_negotiation.cc:407] Connection header received
0925 19:04:39.939308 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.939309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.939322 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.939353 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.939740 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52286: BlockingRecv error: recv got EOF from 127.0.0.1:52286 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 19:04:39.958413 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:39.957355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.0.98.61:38475)
0925 19:04:39.957400 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.957403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.957686 (+ 283us) server_negotiation.cc:407] Connection header received
0925 19:04:39.957792 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.957793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.957806 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.957838 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.958351 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52302: BlockingRecv error: recv got EOF from 127.0.0.1:52302 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 19:04:39.992621 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:39.991805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.0.98.61:38475)
0925 19:04:39.991855 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:39.991857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:39.992009 (+ 152us) server_negotiation.cc:407] Connection header received
0925 19:04:39.992086 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:39.992087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:39.992099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:39.992203 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:39.992549 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52304: BlockingRecv error: recv got EOF from 127.0.0.1:52304 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 19:04:40.058470 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:40.057759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.0.98.61:38475)
0925 19:04:40.057844 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:40.057846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:40.057939 (+ 93us) server_negotiation.cc:407] Connection header received
0925 19:04:40.057974 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:40.057975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:40.057987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:40.058053 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:40.058412 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52316: BlockingRecv error: recv got EOF from 127.0.0.1:52316 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 19:04:40.188916 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:40.187847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.0.98.61:38475)
0925 19:04:40.187922 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:40.187924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:40.188147 (+ 223us) server_negotiation.cc:407] Connection header received
0925 19:04:40.188261 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:40.188263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:40.188296 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:40.188384 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:40.188836 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52324: BlockingRecv error: recv got EOF from 127.0.0.1:52324 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 19:04:40.447070 22773 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:40.446425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52328 (local address 127.0.98.61:38475)
0925 19:04:40.446487 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:40.446491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:40.446506 (+ 15us) server_negotiation.cc:407] Connection header received
0925 19:04:40.446581 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:40.446581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:40.446592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:40.446636 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:40.446955 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52328: BlockingRecv error: recv got EOF from 127.0.0.1:52328 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 19:04:40.793447 22861 catalog_manager.cc:1259] Loaded cluster ID: 95ec9f43b0d448a393512e589fe830cc
I20260925 19:04:40.793524 22861 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf: loading cluster ID for follower catalog manager: success
I20260925 19:04:40.794344 22861 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf: acquiring CA information for follower catalog manager: success
I20260925 19:04:40.794665 22861 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cae1fe31df264002b8114b6832697ecf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:04:40.869504 22904 catalog_manager.cc:1259] Loaded cluster ID: 95ec9f43b0d448a393512e589fe830cc
I20260925 19:04:40.869606 22904 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2: loading cluster ID for follower catalog manager: success
I20260925 19:04:40.870476 22904 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2: acquiring CA information for follower catalog manager: success
I20260925 19:04:40.870812 22904 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ca733d1f8f6470ead3941861045c7b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:04:40.962769 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:40.960153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.0.98.61:38475)
0925 19:04:40.960260 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:40.960262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:40.960311 (+ 49us) server_negotiation.cc:407] Connection header received
0925 19:04:40.960356 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:40.960356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:40.960405 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:40.960484 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:40.960659 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:40.961177 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:40.961893 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:40.962104 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:40.962162 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:40.962359 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:40.962516 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:40.962589 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:40.962627 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 19:04:40.962647 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
I20260925 19:04:40.966815 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:40.964613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.0.98.60:41403)
0925 19:04:40.964745 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:40.964747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:40.964755 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:40.964815 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:40.964815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:40.964847 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:40.964907 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:40.964986 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:40.965432 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:40.966071 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:40.966246 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:40.966366 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:40.966504 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:40.966596 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:40.966656 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:40.966690 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 19:04:40.966706 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260925 19:04:40.971235 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:40.968494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.0.98.60:41403)
0925 19:04:40.968534 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:40.968536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:40.968649 (+ 113us) server_negotiation.cc:407] Connection header received
0925 19:04:40.968712 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:40.968712 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:40.968720 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:40.968771 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:40.968925 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:40.969572 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:40.970489 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:40.970765 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:40.970825 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:40.970948 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:40.970979 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:40.971026 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:40.971116 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 19:04:40.971135 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:04:40.971375 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:40.968606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.0.98.61:38475)
0925 19:04:40.968640 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:40.968666 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:40.968792 (+ 126us) server_negotiation.cc:407] Connection header received
0925 19:04:40.968883 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:40.968884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:40.968894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:40.968936 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:40.969087 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:40.969502 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:40.970544 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:40.970827 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:40.970883 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:40.971022 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:40.971057 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:40.971105 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:40.971283 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 19:04:40.971307 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:04:40.971612 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:40.968462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.0.98.62:39213)
0925 19:04:40.968623 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:40.968625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:40.968764 (+ 139us) server_negotiation.cc:407] Connection header received
0925 19:04:40.968819 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:40.968819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:40.968866 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:40.968941 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:40.969033 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:40.969720 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:40.970712 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:40.971027 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:40.971100 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:40.971267 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:40.971372 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:40.971439 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:40.971498 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 19:04:40.971525 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260925 19:04:40.973321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 22626
I20260925 19:04:40.978463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 22701
I20260925 19:04:40.983199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 22776
2026-09-25T19:04:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2778 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 19:04:41 dist-test-slave-49bz krb5kdc[22991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 19:04:41 dist-test-slave-49bz krb5kdc[22991](info): set up 2 sockets
Sep 25 19:04:41 dist-test-slave-49bz krb5kdc[22991](info): commencing operation
krb5kdc: starting...
W20260925 19:04:41.998497 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/test-user.keytab.
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T19:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T19:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 19:04:42.113215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35383
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35383,127.0.98.61:38465,127.0.98.60:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 19:04:42.192368 23004 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:04:42.192561 23004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:42.192579 23004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:42.192632 23004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:42.193753 23004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:42.193794 23004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:42.193807 23004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:42.193819 23004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:42.193872 23004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:42.195350 23004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35383,127.0.98.61:38465,127.0.98.60:40635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:42.195606 23004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:42.195901 23004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:42.201234 23012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:42.201351 23004 server_base.cc:1034] running on GCE node
W20260925 19:04:42.201252 23010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:42.201248 23009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:42.201808 23004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:42.202121 23004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:42.203291 23004 hybrid_clock.cc:648] HybridClock initialized: now 1790363082203276 us; error 47 us; skew 500 ppm
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:42.205577 23004 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 19:04:42.205979 23004 webserver.cc:460] Webserver started at http://127.0.98.62:33335/ using document root <none> and password file <none>
I20260925 19:04:42.206197 23004 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:42.206276 23004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:42.206449 23004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:42.207484 23004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/data/instance:
uuid: "4d3b5ca394674b8281f91e62fd0d89d9"
format_stamp: "Formatted at 2026-09-25 19:04:42 on dist-test-slave-49bz"
server_key: "1c4902dc12bccd9a68ac773cdb236cf4"
server_key_iv: "31f1cf386ded22c5f34d483c05882d68"
server_key_version: "encryptionkey@0"
I20260925 19:04:42.207860 23004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d3b5ca394674b8281f91e62fd0d89d9"
format_stamp: "Formatted at 2026-09-25 19:04:42 on dist-test-slave-49bz"
server_key: "1c4902dc12bccd9a68ac773cdb236cf4"
server_key_iv: "31f1cf386ded22c5f34d483c05882d68"
server_key_version: "encryptionkey@0"
I20260925 19:04:42.209108 23004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 19:04:42.209774 23019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:42.209942 23004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:42.210038 23004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/wal
uuid: "4d3b5ca394674b8281f91e62fd0d89d9"
format_stamp: "Formatted at 2026-09-25 19:04:42 on dist-test-slave-49bz"
server_key: "1c4902dc12bccd9a68ac773cdb236cf4"
server_key_iv: "31f1cf386ded22c5f34d483c05882d68"
server_key_version: "encryptionkey@0"
I20260925 19:04:42.210119 23004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:42.216882 23004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:42.218570 23004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:42.218714 23004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:42.222324 23004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35383
I20260925 19:04:42.222352 23071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35383 every 8 connection(s)
I20260925 19:04:42.222674 23004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/data/info.pb
I20260925 19:04:42.223577 23072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:42.225067 23072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:42.227439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 23004
I20260925 19:04:42.227558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-0/wal/instance
I20260925 19:04:42.227782 392 external_mini_cluster.cc:1428] Setting key 366328f63896e7b042865d16f10946de
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:42.235275 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.225666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35383 (local address 127.0.0.1:52174)
0925 19:04:42.225814 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.225852 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:42.226328 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:42.226609 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:42.226611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:42.226835 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:42.226979 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.226983 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.228458 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.228459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.229276 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.229279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.229754 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.229756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.229835 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.230408 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:42.230416 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:42.231777 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:42.233817 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.233822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.233824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.233998 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.234329 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.234329 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.234330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.234405 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.234867 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:42.234872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:42.234925 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 19:04:42.235005 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 19:04:42.235129 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260925 19:04:42.235357 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.225738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.0.98.62:35383)
0925 19:04:42.225890 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.225893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.225918 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:04:42.226464 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.226466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.226481 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.226574 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.227090 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.228370 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.229361 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.229649 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.229726 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.231863 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.231871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.231872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.231880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.233737 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.234082 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.234085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.234085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.234205 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.234474 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.234476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.234476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.234790 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.234889 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.235067 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.235218 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260925 19:04:42.236030 23072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:42.236342 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.236231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38465 (local address 127.0.0.1:44302)
0925 19:04:42.236283 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.236324 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 19:04:42.236510 23021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38465: connect: Connection refused (error 111)
I20260925 19:04:42.236936 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.236842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38465 (local address 127.0.0.1:44318)
0925 19:04:42.236889 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.236920 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:04:42.237085 23072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38465: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38465: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 19:04:42.251037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38465
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35383,127.0.98.61:38465,127.0.98.60:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:04:42.286353 23072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } attempt: 1
I20260925 19:04:42.286873 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.286734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38465 (local address 127.0.0.1:44332)
0925 19:04:42.286800 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.286844 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 19:04:42.287237 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.287152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38465 (local address 127.0.0.1:44334)
0925 19:04:42.287196 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.287221 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:04:42.287340 23072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38465: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38465: connect: Connection refused (error 111)
W20260925 19:04:42.330459 23079 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:04:42.330677 23079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:42.330695 23079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:42.330715 23079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:42.331830 23079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:42.331869 23079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:42.331883 23079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:42.331923 23079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:42.331952 23079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:42.333392 23079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35383,127.0.98.61:38465,127.0.98.60:40635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:42.333639 23079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:42.333858 23079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 19:04:42.338822 23079 server_base.cc:1034] running on GCE node
W20260925 19:04:42.338755 23084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:42.338779 23087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:42.338752 23085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:42.339167 23079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:42.339460 23079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:42.340627 23079 hybrid_clock.cc:648] HybridClock initialized: now 1790363082340616 us; error 31 us; skew 500 ppm
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:42.343191 23079 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 19:04:42.343544 23079 webserver.cc:460] Webserver started at http://127.0.98.61:41473/ using document root <none> and password file <none>
I20260925 19:04:42.343739 23079 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:42.343781 23079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:42.343865 23079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:42.344841 23079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/data/instance:
uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d"
format_stamp: "Formatted at 2026-09-25 19:04:42 on dist-test-slave-49bz"
server_key: "f9abfb5a5fd52b4101c58a2380ada2fc"
server_key_iv: "a79247772f32d9e90e2e537e0c9b012d"
server_key_version: "encryptionkey@0"
I20260925 19:04:42.345131 23079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/wal/instance:
uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d"
format_stamp: "Formatted at 2026-09-25 19:04:42 on dist-test-slave-49bz"
server_key: "f9abfb5a5fd52b4101c58a2380ada2fc"
server_key_iv: "a79247772f32d9e90e2e537e0c9b012d"
server_key_version: "encryptionkey@0"
I20260925 19:04:42.346370 23079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 19:04:42.347020 23094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:42.347196 23079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 19:04:42.347258 23079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/wal
uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d"
format_stamp: "Formatted at 2026-09-25 19:04:42 on dist-test-slave-49bz"
server_key: "f9abfb5a5fd52b4101c58a2380ada2fc"
server_key_iv: "a79247772f32d9e90e2e537e0c9b012d"
server_key_version: "encryptionkey@0"
I20260925 19:04:42.347345 23079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:42.355468 23072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } attempt: 2
I20260925 19:04:42.355844 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.355722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38465 (local address 127.0.0.1:44336)
0925 19:04:42.355779 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.355817 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:04:42.356215 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.356129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38465 (local address 127.0.0.1:44348)
0925 19:04:42.356174 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.356199 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:04:42.356321 23072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38465: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38465: connect: Connection refused (error 111)
I20260925 19:04:42.364907 23079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:42.367224 23079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:42.367336 23079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:42.371090 23079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38465
I20260925 19:04:42.371122 23146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38465 every 8 connection(s)
I20260925 19:04:42.371459 23079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/data/info.pb
I20260925 19:04:42.372457 23147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:42.373961 23147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:42.375360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 23079
I20260925 19:04:42.375460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-1/wal/instance
I20260925 19:04:42.375655 392 external_mini_cluster.cc:1428] Setting key d381d17075ff016b2befa009aa8788d6
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:42.383711 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.374692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.0.98.62:35383)
0925 19:04:42.374749 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.374751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.374967 (+ 216us) server_negotiation.cc:407] Connection header received
0925 19:04:42.375637 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.375638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.375649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.375692 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.376533 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.377117 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.378147 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.378499 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.378566 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.380666 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.380668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.380669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.380677 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.382356 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.382758 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.382759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.382760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.382823 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.383052 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.383053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.383053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.383205 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.383282 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.383522 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.383618 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:04:42.383901 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.374661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35383 (local address 127.0.0.1:52182)
0925 19:04:42.374855 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.374890 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:42.375523 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:42.375769 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:42.375773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:42.376154 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:42.376439 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.376446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.377217 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.377218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.378021 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.378024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.378610 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.378611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.378726 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.379244 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:42.379252 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:42.380572 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:42.382449 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.382455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.382457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.382679 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.382890 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.382891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.382891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.382986 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.383274 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:42.383278 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:42.383390 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 19:04:42.383469 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 19:04:42.383602 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260925 19:04:42.384500 23147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 19:04:42.392159 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.384782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.0.98.61:38465)
0925 19:04:42.384937 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.384946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.384960 (+ 14us) server_negotiation.cc:407] Connection header received
0925 19:04:42.384978 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.384980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.384990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.385049 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.385342 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.386099 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.387049 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.387551 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.387638 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.389344 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.389350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.389350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.389356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.390989 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.391396 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.391398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.391399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.391448 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.391662 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.391662 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.391663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.391785 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.391844 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.392018 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.392083 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260925 19:04:42.392323 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.384749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38465 (local address 127.0.0.1:44354)
0925 19:04:42.384799 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.384810 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:42.384859 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:42.385067 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:42.385067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:42.385207 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:42.385253 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.385254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.386172 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.386172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.386963 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.386966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.387631 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.387632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.387683 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.388113 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:42.388116 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:42.389259 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:42.391076 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.391077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.391078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.391307 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.391512 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.391513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.391514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.391595 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.391854 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:42.391854 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:42.391905 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 19:04:42.391997 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 19:04:42.392150 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":47}
I20260925 19:04:42.393146 23147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:42.393509 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.393377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:46908)
0925 19:04:42.393428 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.393480 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 19:04:42.393669 23096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
I20260925 19:04:42.394099 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.393985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:46924)
0925 19:04:42.394043 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.394076 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 19:04:42.394268 23147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40635: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 19:04:42.403093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOuSOt7/build/release/bin/kudu
/tmp/dist-test-taskOuSOt7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40635
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35383,127.0.98.61:38465,127.0.98.60:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 19:04:42.443506 23147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } attempt: 1
I20260925 19:04:42.444063 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.443914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:46938)
0925 19:04:42.443982 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.444029 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 19:04:42.444423 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.444333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:46942)
0925 19:04:42.444376 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.444403 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 19:04:42.444535 23147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40635: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
I20260925 19:04:42.447470 23072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } attempt: 3
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 19:04:42.454402 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.447795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.0.98.61:38465)
0925 19:04:42.447852 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.447854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.447866 (+ 12us) server_negotiation.cc:407] Connection header received
0925 19:04:42.447934 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.447935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.447946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.448012 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.448337 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.448825 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.449518 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.449812 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.449888 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.451980 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.451982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.451982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.451989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.453279 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.453619 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.453620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.453620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.453674 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.453872 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.453873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.453873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.453985 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.454061 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.454217 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.454314 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 19:04:42.454403 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.447729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38465 (local address 127.0.0.1:44360)
0925 19:04:42.447782 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.447794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:42.447867 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:42.448030 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:42.448030 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:42.448189 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:42.448255 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.448257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.448912 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.448914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.449430 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.449431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.449886 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.449887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.449965 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.450472 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:42.450476 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:42.451892 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:42.453374 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.453374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.453375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.453540 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.453741 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.453742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.453742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.453801 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.454052 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:42.454053 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:42.454106 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 19:04:42.454192 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:04:42.454305 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 19:04:42.454845 23072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:42.455173 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.455088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:46948)
0925 19:04:42.455124 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.455149 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 19:04:42.455446 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.455369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:46952)
0925 19:04:42.455408 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.455428 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 19:04:42.455533 23072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40635: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
W20260925 19:04:42.484445 23154 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 19:04:42.484685 23154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 19:04:42.484714 23154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 19:04:42.484757 23154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 19:04:42.485836 23154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 19:04:42.485905 23154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 19:04:42.485931 23154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 19:04:42.485965 23154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 19:04:42.485988 23154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 19:04:42.486670 23072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } attempt: 1
I20260925 19:04:42.487083 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.486947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:46954)
0925 19:04:42.487013 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.487056 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 19:04:42.487403 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.487307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:46960)
0925 19:04:42.487353 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.487380 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 19:04:42.487484 23072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40635: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
I20260925 19:04:42.487803 23154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35383,127.0.98.61:38465,127.0.98.60:40635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 18:08:19 UTC on ccycloud.quasar-jiyrki.root.comops.site
build id 4673100
I20260925 19:04:42.488054 23154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 19:04:42.488283 23154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 19:04:42.492916 23160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:42.492923 23162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 19:04:42.492966 23159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 19:04:42.493227 23154 server_base.cc:1034] running on GCE node
I20260925 19:04:42.493468 23154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 19:04:42.493748 23154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 19:04:42.494884 23154 hybrid_clock.cc:648] HybridClock initialized: now 1790363082494866 us; error 37 us; skew 500 ppm
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 19:04:42.497365 23154 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 19:04:42.497782 23154 webserver.cc:460] Webserver started at http://127.0.98.60:34531/ using document root <none> and password file <none>
I20260925 19:04:42.498003 23154 fs_manager.cc:359] Metadata directory not provided
I20260925 19:04:42.498047 23154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 19:04:42.498208 23154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 19:04:42.499279 23154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/data/instance:
uuid: "b1e9ad1ab5704b62b75c4099d3335d98"
format_stamp: "Formatted at 2026-09-25 19:04:42 on dist-test-slave-49bz"
server_key: "39e37afd2bf37a2daefafa194e6c8771"
server_key_iv: "3b3eddaff009539c36a8a81698123a5a"
server_key_version: "encryptionkey@0"
I20260925 19:04:42.499639 23154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/wal/instance:
uuid: "b1e9ad1ab5704b62b75c4099d3335d98"
format_stamp: "Formatted at 2026-09-25 19:04:42 on dist-test-slave-49bz"
server_key: "39e37afd2bf37a2daefafa194e6c8771"
server_key_iv: "3b3eddaff009539c36a8a81698123a5a"
server_key_version: "encryptionkey@0"
I20260925 19:04:42.500941 23154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 19:04:42.501637 23169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 19:04:42.501821 23154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 19:04:42.501909 23154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/wal
uuid: "b1e9ad1ab5704b62b75c4099d3335d98"
format_stamp: "Formatted at 2026-09-25 19:04:42 on dist-test-slave-49bz"
server_key: "39e37afd2bf37a2daefafa194e6c8771"
server_key_iv: "3b3eddaff009539c36a8a81698123a5a"
server_key_version: "encryptionkey@0"
I20260925 19:04:42.501979 23154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 19:04:42.512705 23147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } attempt: 2
I20260925 19:04:42.513161 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.513023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:46972)
0925 19:04:42.513085 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.513125 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:04:42.513515 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.513420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:46988)
0925 19:04:42.513469 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.513495 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 19:04:42.513626 23147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40635: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40635: connect: Connection refused (error 111)
I20260925 19:04:42.515887 23154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 19:04:42.518180 23154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 19:04:42.518390 23154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 19:04:42.522140 23154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40635
I20260925 19:04:42.522188 23221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40635 every 8 connection(s)
I20260925 19:04:42.522519 23154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/data/info.pb
I20260925 19:04:42.523445 23222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 19:04:42.524981 23222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:42.527727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOuSOt7/build/release/bin/kudu as pid 23154
I20260925 19:04:42.527839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOuSOt7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790362794046639-392-0/minicluster-data/master-2/wal/instance
I20260925 19:04:42.528052 392 external_mini_cluster.cc:1428] Setting key 13c950d701d9500784d0d0336446ad5b
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 19:04:42.534245 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.525666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.0.98.62:35383)
0925 19:04:42.525727 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.525734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.525899 (+ 165us) server_negotiation.cc:407] Connection header received
0925 19:04:42.526382 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.526383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.526391 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.526443 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.526989 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.527443 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.528761 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.528985 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.529036 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.531223 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.531225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.531225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.531232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.532748 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.533252 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.533253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.533254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.533306 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.533590 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.533591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.533592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.533704 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.533780 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.534072 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.534157 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 19:04:42.534437 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.525621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35383 (local address 127.0.0.1:52196)
0925 19:04:42.525775 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.525818 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:42.526293 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:42.526503 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:42.526506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:42.526740 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:42.526924 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.526928 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.527539 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.527540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.528697 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.528699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.529089 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.529091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.529243 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.529820 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:42.529828 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:42.531159 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:42.532853 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.532860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.532862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.533172 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.533376 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.533376 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.533377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.533513 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.533772 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:42.533778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:42.533908 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 19:04:42.534027 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 19:04:42.534184 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260925 19:04:42.535290 23222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:42.537252 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.528421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.0.98.60:40635)
0925 19:04:42.528616 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.528620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.528644 (+ 24us) server_negotiation.cc:407] Connection header received
0925 19:04:42.528707 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.528710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.528730 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.528802 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.529519 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.530378 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.531156 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.531606 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.531673 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.533902 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.533909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.533910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.533917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.535991 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.536383 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.536385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.536386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.536445 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.536642 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.536643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.536643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.536763 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.536899 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.536973 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.537048 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 19:04:42.542526 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.535627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.0.98.61:38465)
0925 19:04:42.535684 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.535686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.535694 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:42.535725 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.535726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.535734 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.535778 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.536107 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.536779 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.537685 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.537977 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.538073 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.539896 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.539897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.539898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.539905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.541384 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.541744 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.541745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.541746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.541802 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.542021 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.542022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.542022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.542138 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.542197 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.542371 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.542441 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 19:04:42.542526 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.535559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38465 (local address 127.0.0.1:44362)
0925 19:04:42.535612 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.535623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:42.535668 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:42.535781 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:42.535782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:42.535986 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:42.536024 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.536025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.536867 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.536868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.537607 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.537609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.538065 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.538066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.538133 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.538503 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:42.538508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:42.539813 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:42.541489 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.541490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.541490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.541661 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.541868 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.541868 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.541869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.541924 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.542199 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:42.542199 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:42.542232 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 19:04:42.542342 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 19:04:42.542432 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 19:04:42.543227 23222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } has no permanent_uuid. Determining permanent_uuid...
I20260925 19:04:42.545225 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.538034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.0.98.62:35383)
0925 19:04:42.538073 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.538075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.538084 (+ 9us) server_negotiation.cc:407] Connection header received
0925 19:04:42.538105 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.538106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.538115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.538164 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.538745 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.539356 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.540073 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.540273 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.540357 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.542628 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.542629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.542630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.542636 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.544079 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.544415 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.544415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.544416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.544470 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.544674 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.544675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.544675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.544780 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.544958 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.544993 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.545061 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 19:04:42.549796 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.543610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:47004)
0925 19:04:42.543662 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.543672 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:42.543707 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:42.543834 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:42.543834 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:42.543936 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:42.543957 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.543958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.544548 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.544550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.544991 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.544992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.545432 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.545433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.545475 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.545920 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:42.545925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:42.547062 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:42.548670 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.548671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.548672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.548887 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.549100 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.549101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.549102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.549166 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.549445 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:42.549445 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:42.549496 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 19:04:42.549580 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 19:04:42.549672 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 19:04:42.549896 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.543680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.0.98.60:40635)
0925 19:04:42.543712 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.543714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.543730 (+ 16us) server_negotiation.cc:407] Connection header received
0925 19:04:42.543747 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.543748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.543758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.543799 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.544027 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.544471 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.545106 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.545336 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.545393 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.547140 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.547142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.547142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.547149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.548594 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.548966 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.548967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.548967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.549030 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.549233 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.549234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.549234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.549378 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.549429 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.549647 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.549765 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":19}
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 19:04:42.551514 23222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98: Bootstrap starting.
I20260925 19:04:42.552639 23222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:42.552959 23222 log.cc:824] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:42.553862 23222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98: No bootstrap required, opened a new log
I20260925 19:04:42.553900 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.545859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.0.98.61:38465)
0925 19:04:42.545900 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.545901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.545926 (+ 25us) server_negotiation.cc:407] Connection header received
0925 19:04:42.545946 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.545947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.545957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.546016 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.546661 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.547230 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.548135 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.548449 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.548566 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.551169 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.551171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.551171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.551178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.552735 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.553139 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.553140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.553140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.553217 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.553429 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.553430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.553430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.553564 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.553687 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.553782 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.553841 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 19:04:42.555461 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 19:04:42.555879 23222 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } }
I20260925 19:04:42.556005 23222 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9ad1ab5704b62b75c4099d3335d98, State: Initialized, Role: FOLLOWER
I20260925 19:04:42.556092 23222 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } }
I20260925 19:04:42.556373 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } } }
I20260925 19:04:42.556435 23222 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:42.556489 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:42.556579 23046 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4d3b5ca394674b8281f91e62fd0d89d9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52204
W20260925 19:04:42.558240 23239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:04:42.558318 23239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:04:42.558626 23154 master_runner.cc:186] Error getting master registration for 127.0.98.62:35383: OK, instance_id { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" instance_seqno: 1790363082221290 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:04:42.561401 23154 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:04:42.562618 23072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } attempt: 2
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 19:04:42.570456 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.562863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:47010)
0925 19:04:42.562909 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.562919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:42.562953 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:42.563176 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:42.563177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:42.563349 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:42.563396 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.563398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.564267 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.564268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.565069 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.565072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.565528 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.565529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.565603 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.566147 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:42.566153 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:42.567624 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:42.569403 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.569404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.569405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.569580 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.569789 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.569789 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.569790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.569851 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.570126 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:42.570127 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:42.570171 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 19:04:42.570254 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 19:04:42.570363 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 19:04:42.570454 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.562977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.0.98.60:40635)
0925 19:04:42.563029 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.563031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.563039 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:42.563069 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.563069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.563092 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.563153 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.563470 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.564181 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.565153 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.565450 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.565520 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.567713 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.567715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.567715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.567722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.569324 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.569652 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.569654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.569654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.569706 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.569912 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.569913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.569914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.570043 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.570123 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.570299 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.570367 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 19:04:42.571635 23072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9: Bootstrap starting.
I20260925 19:04:42.572234 23072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:42.572518 23072 log.cc:824] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:42.573370 23072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9: No bootstrap required, opened a new log
I20260925 19:04:42.574759 23072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } }
I20260925 19:04:42.574890 23072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3b5ca394674b8281f91e62fd0d89d9, State: Initialized, Role: FOLLOWER
I20260925 19:04:42.574971 23072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } }
I20260925 19:04:42.575284 23072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:42.575346 23046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 19:04:42.575433 23046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:42.576211 23046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } }
I20260925 19:04:42.576442 23046 leader_election.cc:288] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [CANDIDATE]: Term 1 election: Requested vote from peers ca7ce994a5d14e20977f7f1c2ce38a9d (127.0.98.61:38465), b1e9ad1ab5704b62b75c4099d3335d98 (127.0.98.60:40635)
I20260925 19:04:42.575248 23242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } } }
I20260925 19:04:42.576918 23242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 19:04:42.577502 23253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:04:42.577551 23253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:04:42.577610 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b1e9ad1ab5704b62b75c4099d3335d98"
I20260925 19:04:42.577733 23196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:42.578629 23196 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3b5ca394674b8281f91e62fd0d89d9 in term 1.
I20260925 19:04:42.578835 23021 leader_election.cc:302] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3b5ca394674b8281f91e62fd0d89d9, b1e9ad1ab5704b62b75c4099d3335d98; no voters:
I20260925 19:04:42.578954 23242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 19:04:42.579071 23242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3b5ca394674b8281f91e62fd0d89d9, State: Running, Role: LEADER
I20260925 19:04:42.579145 23242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } }
I20260925 19:04:42.579571 23242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3b5ca394674b8281f91e62fd0d89d9. Latest consensus state: current_term: 1 leader_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } } }
I20260925 19:04:42.579651 23242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:42.579810 23261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 19:04:42.579958 23261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 19:04:42.578564 23004 master_runner.cc:186] Error getting master registration for 127.0.98.61:38465: OK, instance_id { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" instance_seqno: 1790363082369967 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 19:04:42.580832 23004 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 19:04:42.581845 23196 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [term 1 FOLLOWER]: Refusing update from remote peer 4d3b5ca394674b8281f91e62fd0d89d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:42.582337 23259 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 19:04:42.583746 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3b5ca394674b8281f91e62fd0d89d9. Latest consensus state: current_term: 1 leader_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } } }
I20260925 19:04:42.583834 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:42.592105 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.581995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.0.98.62:35383)
0925 19:04:42.582066 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.582068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.582211 (+ 143us) server_negotiation.cc:407] Connection header received
0925 19:04:42.582406 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.582420 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.582433 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.582468 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.583785 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.584488 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.586228 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.586555 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.586598 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.588826 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.588827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.588828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.588834 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.590463 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.591022 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.591023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.591024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.591087 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.591394 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.591395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.591396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.591530 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.591620 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.591932 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.592007 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 19:04:42.594589 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.582212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.0.98.61:38465)
0925 19:04:42.582285 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.582287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.584051 (+ 1764us) server_negotiation.cc:407] Connection header received
0925 19:04:42.584176 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.584178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.584188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.584217 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.585303 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.586022 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.587847 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.588174 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.588214 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.590783 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.590784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.590785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.590802 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.592967 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.593521 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.593522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.593523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.593597 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.593900 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.593901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.593902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.594036 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.594125 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.594417 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.594494 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:04:42.595544 23242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } } }
I20260925 19:04:42.595643 23242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:42.595741 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } } }
I20260925 19:04:42.595784 23262 mvcc.cc:205] Tried to move back new op lower bound from 7333327186253369344 to 7333327186245087232. Current Snapshot: MvccSnapshot[applied={T|T < 7333327186253369344}]
I20260925 19:04:42.595821 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:42.598948 23261 catalog_manager.cc:1347] Generated new cluster ID: ee191bb0ab134bf1ba4750dbb7a3a992
I20260925 19:04:42.598994 23261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 19:04:42.604727 23147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } attempt: 3
I20260925 19:04:42.615923 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.582375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.0.98.60:40635)
0925 19:04:42.582408 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.582410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.593133 (+ 10723us) server_negotiation.cc:407] Connection header received
0925 19:04:42.593281 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.593282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.593304 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.593333 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.594140 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.594879 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.597097 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.597388 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.597425 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.598821 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.598822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.598823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.598829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.600580 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.614684 (+ 14104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.614686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.614686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.614762 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.615104 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.615119 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.615119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.615263 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.615358 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.615730 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.615821 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 19:04:42 dist-test-slave-49bz krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790363082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 19:04:42.630868 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.604962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40635 (local address 127.0.0.1:47022)
0925 19:04:42.605002 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 19:04:42.605014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 19:04:42.605069 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 19:04:42.610492 (+ 5423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 19:04:42.610493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 19:04:42.610687 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 19:04:42.610733 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.610735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.623152 (+ 12417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.623153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.623998 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 19:04:42.624001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.624547 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.624548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 19:04:42.624609 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.625127 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 19:04:42.625133 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 19:04:42.627432 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 19:04:42.629465 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.629466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.629467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.629681 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.629882 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 19:04:42.629883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 19:04:42.629883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 19:04:42.629942 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 19:04:42.630234 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 19:04:42.630235 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 19:04:42.630574 (+ 339us) client_negotiation.cc:769] Sending connection context
0925 19:04:42.630660 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 19:04:42.630763 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 19:04:42.631574 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:42.605061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47022 (local address 127.0.98.60:40635)
0925 19:04:42.610328 (+ 5267us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.610330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.610343 (+ 13us) server_negotiation.cc:407] Connection header received
0925 19:04:42.610368 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.610369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.610380 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.610457 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.622293 (+ 11836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.623069 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.624082 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:42.624424 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:42.624485 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:42.627533 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 19:04:42.627534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 19:04:42.627535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 19:04:42.627542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 19:04:42.629373 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.629751 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.629752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.629752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.629810 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 19:04:42.630006 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 19:04:42.630007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 19:04:42.630007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 19:04:42.630147 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 19:04:42.630211 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:42.631208 (+ 997us) server_negotiation.cc:299] Negotiation successful
0925 19:04:42.631291 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5242,"thread_start_us":89,"threads_started":1}
I20260925 19:04:42.631559 23261 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 19:04:42.633415 23147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d: Bootstrap starting.
I20260925 19:04:42.633456 23261 catalog_manager.cc:1504] Loading token signing keys...
I20260925 19:04:42.634119 23147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d: Neither blocks nor log segments found. Creating new log.
I20260925 19:04:42.634418 23147 log.cc:824] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d: Log is configured to *not* fsync() on all Append() calls
I20260925 19:04:42.635489 23147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d: No bootstrap required, opened a new log
I20260925 19:04:42.637439 23147 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } }
I20260925 19:04:42.637566 23147 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7ce994a5d14e20977f7f1c2ce38a9d, State: Initialized, Role: FOLLOWER
I20260925 19:04:42.637638 23147 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } }
I20260925 19:04:42.637975 23147 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 19:04:42.638021 23121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d"
I20260925 19:04:42.638139 23121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [term 0 FOLLOWER]: Advancing to term 1
I20260925 19:04:42.638985 23121 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3b5ca394674b8281f91e62fd0d89d9 in term 1.
I20260925 19:04:42.639183 23120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [term 1 FOLLOWER]: Refusing update from remote peer 4d3b5ca394674b8281f91e62fd0d89d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 19:04:42.639442 23242 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 19:04:42.640223 23283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 19:04:42.640344 23283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 19:04:42.640709 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } } }
I20260925 19:04:42.640836 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:42.642333 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3b5ca394674b8281f91e62fd0d89d9. Latest consensus state: current_term: 1 leader_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } } }
I20260925 19:04:42.642421 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:42.642457 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } } }
I20260925 19:04:42.642503 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 19:04:42.644688 23261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9: Generated new TSK 0
I20260925 19:04:42.644856 23261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 19:04:42.647046 23259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b5ca394674b8281f91e62fd0d89d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35383 } } peers { permanent_uuid: "ca7ce994a5d14e20977f7f1c2ce38a9d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38465 } } peers { permanent_uuid: "b1e9ad1ab5704b62b75c4099d3335d98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40635 } } }
I20260925 19:04:42.647138 23259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3b5ca394674b8281f91e62fd0d89d9 [sys.catalog]: This master's current role is: LEADER
I20260925 19:04:42.648164 23285 mvcc.cc:205] Tried to move back new op lower bound from 7333327186453209088 to 7333327186245087232. Current Snapshot: MvccSnapshot[applied={T|T < 7333327186253369344}]
W20260925 19:04:42.671797 23075 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.669951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.0.98.62:35383)
0925 19:04:42.669998 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.670000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.671063 (+ 1063us) server_negotiation.cc:407] Connection header received
0925 19:04:42.671183 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.671184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.671196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.671227 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.671724 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52220: BlockingRecv error: recv got EOF from 127.0.0.1:52220 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 19:04:42.672614 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.670437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.0.98.61:38465)
0925 19:04:42.670477 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.670479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.672047 (+ 1568us) server_negotiation.cc:407] Connection header received
0925 19:04:42.672148 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.672149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.672161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.672196 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.672549 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44382: BlockingRecv error: recv got EOF from 127.0.0.1:44382 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 19:04:42.673357 23270 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.670764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.0.98.60:40635)
0925 19:04:42.670807 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.670809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.672851 (+ 2042us) server_negotiation.cc:407] Connection header received
0925 19:04:42.672949 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.672950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.672962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.672992 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.673298 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47034: BlockingRecv error: recv got EOF from 127.0.0.1:47034 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 19:04:42.675766 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.674950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.0.98.61:38465)
0925 19:04:42.674986 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.674987 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.675178 (+ 191us) server_negotiation.cc:407] Connection header received
0925 19:04:42.675293 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.675294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.675313 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.675343 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.675705 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44384: BlockingRecv error: recv got EOF from 127.0.0.1:44384 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 19:04:42.678990 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.678025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.0.98.61:38465)
0925 19:04:42.678160 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.678162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.678451 (+ 289us) server_negotiation.cc:407] Connection header received
0925 19:04:42.678570 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.678571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.678582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.678612 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.678942 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44396: BlockingRecv error: recv got EOF from 127.0.0.1:44396 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260925 19:04:42.683183 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.682192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.0.98.61:38465)
0925 19:04:42.682226 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.682228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.682571 (+ 343us) server_negotiation.cc:407] Connection header received
0925 19:04:42.682756 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.682758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.682769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.682799 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.683137 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44408: BlockingRecv error: recv got EOF from 127.0.0.1:44408 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 19:04:42.689199 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.688366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.0.98.61:38465)
0925 19:04:42.688403 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.688404 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.688660 (+ 256us) server_negotiation.cc:407] Connection header received
0925 19:04:42.688766 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.688767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.688778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.688808 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.689149 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44416: BlockingRecv error: recv got EOF from 127.0.0.1:44416 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 19:04:42.699433 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.698587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.0.98.61:38465)
0925 19:04:42.698626 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.698627 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.698874 (+ 247us) server_negotiation.cc:407] Connection header received
0925 19:04:42.698975 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.698977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.698988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.699017 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.699376 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44426: BlockingRecv error: recv got EOF from 127.0.0.1:44426 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 19:04:42.717795 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.716851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.0.98.61:38465)
0925 19:04:42.716905 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.716907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.717193 (+ 286us) server_negotiation.cc:407] Connection header received
0925 19:04:42.717299 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.717300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.717312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.717342 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.717736 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44440: BlockingRecv error: recv got EOF from 127.0.0.1:44440 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 19:04:42.751976 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.751183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.0.98.61:38465)
0925 19:04:42.751226 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.751228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.751377 (+ 149us) server_negotiation.cc:407] Connection header received
0925 19:04:42.751448 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.751449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.751460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.751560 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.751897 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44454: BlockingRecv error: recv got EOF from 127.0.0.1:44454 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 19:04:42.817838 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.817110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.0.98.61:38465)
0925 19:04:42.817165 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.817167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.817290 (+ 123us) server_negotiation.cc:407] Connection header received
0925 19:04:42.817327 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.817329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.817342 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.817418 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.817779 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44464: BlockingRecv error: recv got EOF from 127.0.0.1:44464 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 19:04:42.947566 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:42.946916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.0.98.61:38465)
0925 19:04:42.946990 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:42.946992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:42.947087 (+ 95us) server_negotiation.cc:407] Connection header received
0925 19:04:42.947132 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:42.947132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:42.947143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:42.947203 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:42.947513 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44468: BlockingRecv error: recv got EOF from 127.0.0.1:44468 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 19:04:43.205459 23151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 19:04:43.204821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.0.98.61:38465)
0925 19:04:43.204863 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:43.204865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:43.204916 (+ 51us) server_negotiation.cc:407] Connection header received
0925 19:04:43.204995 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:43.204996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:43.205006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:43.205067 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 19:04:43.205396 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44478: BlockingRecv error: recv got EOF from 127.0.0.1:44478 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 19:04:43.558738 23239 catalog_manager.cc:1259] Loaded cluster ID: ee191bb0ab134bf1ba4750dbb7a3a992
I20260925 19:04:43.558856 23239 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98: loading cluster ID for follower catalog manager: success
I20260925 19:04:43.559674 23239 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98: acquiring CA information for follower catalog manager: success
I20260925 19:04:43.559983 23239 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b1e9ad1ab5704b62b75c4099d3335d98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:04:43.640810 23283 catalog_manager.cc:1259] Loaded cluster ID: ee191bb0ab134bf1ba4750dbb7a3a992
I20260925 19:04:43.640921 23283 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d: loading cluster ID for follower catalog manager: success
I20260925 19:04:43.641738 23283 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d: acquiring CA information for follower catalog manager: success
I20260925 19:04:43.642063 23283 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca7ce994a5d14e20977f7f1c2ce38a9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 19:04:43.721168 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:43.718613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.0.98.61:38465)
0925 19:04:43.718753 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:43.718755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:43.718766 (+ 11us) server_negotiation.cc:407] Connection header received
0925 19:04:43.718827 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:43.718827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:43.718889 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:43.718992 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:43.719137 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:43.719593 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:43.720319 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:43.720539 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:43.720613 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:43.720793 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:43.720902 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:43.721016 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:43.721039 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 19:04:43.721061 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":54,"threads_started":1}
I20260925 19:04:43.725127 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:43.722975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.0.98.60:40635)
0925 19:04:43.723064 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:43.723066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:43.723074 (+ 8us) server_negotiation.cc:407] Connection header received
0925 19:04:43.723156 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:43.723156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:43.723189 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:43.723249 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:43.723338 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:43.723797 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:43.724490 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:43.724665 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:43.724713 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:43.724823 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:43.724917 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:43.724969 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:43.725000 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 19:04:43.725014 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
I20260925 19:04:43.729660 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:43.726943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.0.98.61:38465)
0925 19:04:43.726987 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:43.726989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:43.727083 (+ 94us) server_negotiation.cc:407] Connection header received
0925 19:04:43.727209 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:43.727210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:43.727222 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:43.727274 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:43.727380 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:43.727897 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:43.728894 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:43.729169 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:43.729222 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:43.729387 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:43.729416 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:43.729442 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:43.729547 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 19:04:43.729575 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 19:04:43.729768 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:43.726789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.0.98.62:35383)
0925 19:04:43.726903 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:43.726905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:43.726928 (+ 23us) server_negotiation.cc:407] Connection header received
0925 19:04:43.727039 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:43.727041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:43.727091 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:43.727161 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:43.727317 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:43.728071 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:43.728848 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:43.729138 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:43.729210 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:43.729347 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:43.729452 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:43.729507 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:43.729649 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 19:04:43.729676 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
I20260925 19:04:43.730139 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 19:04:43.727155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.0.98.60:40635)
0925 19:04:43.727198 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 19:04:43.727200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 19:04:43.727231 (+ 31us) server_negotiation.cc:407] Connection header received
0925 19:04:43.727328 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 19:04:43.727329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 19:04:43.727338 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 19:04:43.727384 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 19:04:43.727551 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:43.728205 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:43.729119 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 19:04:43.729396 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 19:04:43.729467 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 19:04:43.729620 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 19:04:43.729649 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 19:04:43.729675 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0925 19:04:43.730046 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 19:04:43.730070 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 19:04:43.737579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 23004
I20260925 19:04:43.743127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 23079
I20260925 19:04:43.748080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOuSOt7/build/release/bin/kudu with pid 23154
2026-09-25T19:04:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2764 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5543 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (289708 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS